Kafka Dashboard

Kafka is a high performance, highly fault-tolerant, and highly scalable, real-time messaging system. Pulse provides you with separate windows to view insights on the following components of Kafka:

  1. Kafka streaming details
  2. Topics
  3. Brokers
  4. Consumers

Click Kafka --> Dashboard in the left pane to access the Kafka dashboard. The Dashboard lets you monitor Kafka based clusters in detail.

The following metrics are displayed in the summary panel.

MetricDescription
BrokersNumber of brokers or Kafka servers in the Kafka cluster.
Active ControllerNumber of controllers currently active in the kafka cluster. One broker is elected as the active controller within the brokers group. To know which broker is the active controller, click the number in the Active Controller metric.
Active TopicsNumber of currently active topics or active stream of records.
PartitionsNumber of partitions of data the Kafka topics are divided into.
Online PartitionsNumber of partitions that are currently active or online.
Under Replicated PartitionsNumber of partitions that are yet to be replicated.

Kafka Streaming Details

Kafka streaming lets you analyze data stored in your Kafka cluster to monitor the distribution of data streams. Data is distributed across the following partitions.

  • Producer: A producer is the source of data in your Kafka cluster that sends the data to multiple topics in the broker. You can view the topics to where a producer streams data.
  • Topic: A category where streaming and publishing data is stored. Multiple streaming data can be stored and can be published to one or more channels or consumers.
  • Consumer: A category where published data from topics is read. One category can read published data from multiple topics.

The following image is a representation of Kafka Streaming Details.

Streaming Details

To view the distribution of data to or from a single unit end-to-end, click the element. You can then see all incoming and outgoing streams. You can also search by name by typing the element name in the search box. This displays the list of streaming and publishing pipelines.

Events

Events are any occurrences of a stream or message in a pipeline. You can monitor the internal events in the Kafka ecosystem for a faster throughput and better stability in a data pipeline. There can be multiple events in a day that you can observe as shown in the following image.

Events

Other Charts on the Kafka Dashboard

MetricDescription
Top Brokers by RequestDisplays the top brokers with maximum requests per second on nodes connected to the cluster.
Top Topics by SizeDisplays the topics of maximum sizes.
Top Partitions by SizeDisplays partitions of maximum sizes in a cluster and can be filtered by a specific topic. To filter out partitions by a topic of your choice, click Filter by topic drop down and select the topic.
Top Topics with SkewDisplays the brokers with maximum skewness on a topic.
Top Consumer Group by LagDisplays the consumer group with highest lag while receiving data from producer. A consumer group might not be able to receive data at the same rate as the producer streams. Click the group to know more details about the lag. You can also filter by consumer group. Click the Consumer Group drop down and select a group.