YARN Capacity
The YARN Capacity dashboard gives you information about memory utilized by jobs of different application types in a selected queue for computing. To access YARN Capacity dashboard, click YARN -> Dashboard in the left pane.
note
The default time range is Last 24 hrs and CST timezone. To view statistics from a custom date range, click the icon and select a time frame and timezone.
Summary panel
The top panels of the YARN dashboard display the following states and status of applications.
State | Description |
---|---|
New | The application that is just created. |
New_Saving | The application that is being saved. |
Submitted | The application that is just submitted. |
Accepted | The application accepted to run by the YARN scheduler. |
Running | The application that is currently running. |
Finished | The application that completed execution. |
Failed | The application that failed. |
Killed | The application that you terminated. |
Status | Description |
---|---|
Succeeded | The number of jobs that completed successfully. |
Failed | The number of jobs that failed. |
Killed | The number of jobs that were forcefully or unexpectedly terminated. |
Undefined | The number of jobs that have a state other than Finished. |
The top right panel of the YARN dashboard displays the number of jobs running in the queue by the following applications.
note
To view details of specific job types, click the number under these metrics.
Application Type | Description |
---|---|
YARN | The number of YARN applications submitted in the queue. |
Tez | The number of Tez applications submitted in the queue. |
Spark | The number of Spark applications submitted in the queue. |
MapReduce | The number of jobs running in MapReduce application in the queue. |
Queues
In Queues tab, you can see the root queue, default queue, and custom queue(s) defined by the cluster administrator.
- root: A predefined queue of YARN. This queue is a parent of the queues available in your cluster. This queue uses 100% of resources.
- default: A designated queue defined by the administrator. This queue contains jobs that do not have a queue allocated.
note
To view memory capacity allocated to or used by resources on a queue, click the queue in the Queues tab. You can then view the statistics on the YARN Capacity dashboard.
Node Labels
The Node Labels displays the label name of the node partition and metrics of the nodes having similar characteristics. By default, the nodes belong to the Default label. For every node label, you can view the queue, vcore, and memory metrics.
To know more about the statistics of a metric, click the number under these metrics.
Queue Metric Distributions
The Query Metric Distributions Sankey chart displays the flow of data values from Queues to Application Type to Jobs.
Reading a Sankey Chart
The following screenshot is an example of a Queue Metric Distributions Sankey chart displayed by Duration.
You can gather the following information from the chart.
note
To see the distribution in numbers, hover over the Sankey chart.
You can observe the following in Queues.
- 62.95% of the time is spent by applications running on the
default
queue. - 36.58% of the time is spent by applications running on the
spark_jobs_q
queue. - The remaining 0.48% of the time is spent by applications running on the
llap
queue.
The applications in these queues are either of the following types.
- Spark
- MapReduce
- Tez
- Slider
You can observe the following based on the application types.
- 92.4% of jobs are done by Spark application.
- 3.33% of jobs are done by MapReduce application.
- 4.04% of jobs are done by Tez application.
- 0.24% of jobs are done by Slider jobs.
Out of the jobs submitted, you can observe the following.
- 99.05% of jobs were completed within an average time of 6.00 milliseconds to 1.85 hours.
- The remaining jobs were completed within an average of 1.65 days to 3.13 days.
Viewing Sankey chart by distribution
You can view the Sankey chart by the following distributions.
- Duration
- Duration Variance
- VCore Seconds
- Memory Seconds
Other Capacity Charts
The following table gives you an overview of the other metrics charts in Capacity.
note
The default time range is Last 24 hrs. To view statistics from a custom date range, click the icon and select a time frame and timezone of your choice.
Chart Name | Description |
---|---|
Queue Distribution | Distribution of queue capacity in percentage, configured in the cluster. You can see the capacity of root, default, and other custom defined queues. |
Queue Usage | % of resource usage in a queue, calculated in percentage, in a timeframe. The queue usage lets you identify when resources in a queue are consumed at peak. |
Memory Usage | Amount of memory used within the queue in the cluster in a particular timeframe. |
VCore Usage | Number of virtual cores used within the queue in the cluster. |
VCore Usage Per User | Number of virtual cores used by an individual user. |
Memory Usage Per Usage | Amount of memory used per user. |
Memory Utilization | Amount of memory utilized within the queue. You can monitor the following memory statistics. Peak mem utilization: The maximum amount of memory used by a queue in a timeframe. The peak memory lets you know when a queue is highly utilized. Avg mem utilization: The average amount of memory used by a queue in a timeframe. Max mem limit: The maximum amount of memory configured according to selected queue. Mem limit: The maximum amount of memory defined by the administrator for a queue. |
No. of Containers | The number of containers allocated to run a single or multiple applications in a queue in a timeframe. The following metrics are used in monitoring the containers. Max containers allocated: The maximum number of containers allocated to a queue. Avg containers allocated: The average number of containers allocated to a queue. |