Spark Batch Details

Batch Details panel displays the details of batches created by live input data streams that Spark streaming receives. You can view job details of every batch created. The following metrics are displayed in the top panel.

MetricDescription
Batch DurationThe total time taken to complete processing jobs in a batch.
Processing TimeThe time taken to process data streaming jobs in the batch.
Scheduling DelayThe time taken by the scheduler to submit the batch jobs.
Total DelayThe total time taken (Scheduling Delay + Processing Time) by the scheduler to submit the batch jobs.

Batch Output Operators

The Batch Output Operators panel displays data operations that can be pushed externally to database systems or file systems. The following metrics are displayed in this panel. Note: Click on any operation in the list to know more about that batch output operator.

MetricDescription
Output Op IdThe ID of the output operation.
NameThe name of the output operation.
StatusThe final status of the operation.
Job IdsThe ID of jobs the batch output operation is processing.
DurationThe duration of the jobs in that batch operation.
ErrorThe error code of the error that ocurred in that batch output operation.

Within the output operator, you can view following details of stage IDs.

  • Description: The description of the tasks in the stage.
  • Status: The final status of the stage, whether Succeeded or Failed.
  • Time Taken: The time taken to complete processing the stage.

Metrics

Metric GroupMetric NameDescription
ShuffleShuffle ReadAmount of shuffling data read (in bytes).
Shuffle Read RecordsNumber of records of shuffle read.
Shuffle WriteAmount of shuffling data written (in bytes).
Shuffle Write RecordsNumber of records of shuffle write.
CPUExecutor CPU TimeTotal CPU time taken by the executor to run the task.
Executor Run TimeTime taken by the executor to run the task.
DiscInput BytesThe amount of input bytes read during the task.
Input RecordsThe number of input records read.
Output BytesThe amount of output bytes written during the task.
Output RecordsThe number of output records written.
OtherNumber Of TasksThe number of tasks in the stage
Complete TasksThe number of completed tasks.
Active IndicesThe number of indices currently running in the stage.
Completed IndicesThe number of indices that completed execution.
Failed TasksThe number of tasks that failed execution.
Killed TasksThe number of tasks that terminated.
Disk Bytes SpilledThe amount of deserialized form of data on the disk at the time the data is spilt.
Memory Bytes SpilledThe amount of deserialized form of data in memory at the time the data is spilt.

DAG

The Direct Acyclic Graph (DAG) displays a flow diagram of the Spark job.

DAG is a work scheduling graph with finite elements connected in edges and vertices. These elements are also called RDDs (Resilient Distributed Datasets). The RDDs are fault-tolerant in nature.

The order of execution of the jobs in DAG is specified by the directions of the edges in the graph. The graph is acyclic as it has no loops or cycles.

DAG

Task Distribution

The task distribution tab displays how tasks are distributed across the following metrics over percentile values.

MetricDescription
DurationTime taken by the stages to complete.
Scheduler DelayThe waiting time of the task to be scheduled for execution.
Task Deserialize TimeTime taken to deserialize tasks.
Gc TimeTime spent by the JVM in garbage collection while executing a task.
Result SerializationTimeTime spent to serialize a task result.
Getting Result TimeThe time taken by the driver to collect task results.
Peak Execution MemoryThe memory used during shuffles, aggregations, and joins by internal data structures.