Capacity Management

Components

The following components are present in the Yarn Capacity Analyser. These are based on the needs of enterprises during the planning and subsequent operational phases.

alt-text

Calendar Date Selection

Select the Calendar interval from the top-right of the screen. Acceldata stores historical capacity utilization for a configurable duration.

alt-text

Queue Filtering

On the left pane of navigation, the queues can be selected. The overall cluster view is then filtered and information related to the queue is only shown. Currently the default queue is slected.

Child queues are also represented, along with their hierarchies upon selection of specific queues.

alt-text

Application Status Summary

At the top of the page one can see the status of the applications in various stages of their lifecycle. The following stages are displayed:

  • Succeeded
  • Failed
  • Killed
  • Undefined

Additionally, the count of applications of various types with their cumulative numbers is displayed:

  • Yarn
  • Spark
  • Tez
  • MR

Query Metric Distribution

Queue Usage

The usage of the queue from a historical perspective is displayed as well. The total available capacity along with the used capacity is displayed. Also represented is the relationship between parent and child queues, accounting for the hierarchies as well.

Memory & VCore Usage

The following is the representation of usage of Memory and VCore during the selected calendar interval, represented by vertical bar charts.

alt-text

Top K Users (By Memory & VCore)

Top K Users by way of resource consumption is represented in the bar charts. Some users need greater resources than others, which occasionally represents trouble as well.

alt-text

CPU, Memory Utilization & Container Allocation

The following graphs represent the overall CPU & Memory utilization. This visualization is very useful in presenting the state of resources in terms of shortage or wastage.

alt-text

Resource Pool Selection

Resource pools are separated in order to allow different kinds of workloads to be submitted on different queues in question. Typically enterprises use these resource pools for:

  • Ingestion
  • Access

Specific monitoring of capacity on these resource pools is available.