Kafka is a high performance, highly fault-tolerant, and highly scalable, real-time messaging system. Pulse provides you with separate windows to view insights on the following components of Kafka:
- Kafka streaming details
Click Kafka --> Dashboard in the left pane to access the Kafka dashboard. The Dashboard lets you monitor Kafka based clusters in detail.
The following metrics are displayed in the summary panel.
|Brokers||Number of brokers or Kafka servers in the Kafka cluster.|
|Active Controller||Number of controllers currently active in the kafka cluster. One broker is elected as the active controller within the brokers group. To know which broker is the active controller, click the number in the Active Controller metric.|
|Active Topics||Number of currently active topics or active stream of records.|
|Partitions||Number of partitions of data the Kafka topics are divided into.|
|Online Partitions||Number of partitions that are currently active or online.|
|Under Replicated Partitions||Number of partitions that are yet to be replicated.|
Kafka Streaming Details
Kafka streaming lets you analyze data stored in your Kafka cluster to monitor the distribution of data streams. Data is distributed across the following partitions.
- Producer: A producer is the source of data in your Kafka cluster that sends the data to multiple topics in the broker. You can view the topics to where a producer streams data.
- Topic: A category where streaming and publishing data is stored. Multiple streaming data can be stored and can be published to one or more channels or consumers.
- Consumer: A category where published data from topics is read. One category can read published data from multiple topics.
The following image is a representation of Kafka Streaming Details.
To view the distribution of data to or from a single unit end-to-end, click the element. You can then see all incoming and outgoing streams. You can also search by name by typing the element name in the search box. This displays the list of streaming and publishing pipelines.
Events are any occurrences of a stream or message in a pipeline. You can monitor the internal events in the Kafka ecosystem for a faster throughput and better stability in a data pipeline. There can be multiple events in a day that you can observe as shown in the following image.
Other Charts on the Kafka Dashboard
|Top Brokers by Request||Displays the top brokers with maximum requests per second on nodes connected to the cluster.|
|Top Topics by Size||Displays the topics of maximum sizes.|
|Top Partitions by Size||Displays partitions of maximum sizes in a cluster and can be filtered by a specific topic. To filter out partitions by a topic of your choice, click Filter by topic drop down and select the topic.|
|Top Topics with Skew||Displays the brokers with maximum skewness on a topic.|
|Top Consumer Group by Lag||Displays the consumer group with highest lag while receiving data from producer. A consumer group might not be able to receive data at the same rate as the producer streams. Click the group to know more details about the lag. You can also filter by consumer group. Click the Consumer Group drop down and select a group.|