HDFS Dashboard

Enabling HDFS monitoring on Pulse gives you an in-depth overview of the Hadoop file system within your cluster. You can also observe and monitor the overall health of your HDFS cluster pertaining to the CPU and memory usage.

Note: The statistics displayed are from the last 24 hours by default. To view statistics from a custom date range, click the Timeframe icon in the top-right corner of the page and select the required time frame and timezone.

The top panel of the dashboard displays the following metrics.

NameDescription
Name Nodes ActiveThe number of active name nodes in the cluster.
Num Live Data NodesThe number of data nodes that are currently live.
Num Dead Data NodesThe number of data nodes in a dead state.
Capacity RemainingThe amount of HDFS storage available on the cluster, in MB. This is calculated as Capacity Total - (Capacity Used + Capacity used non DFS).
Capacity TotalThe total HDFS storage on the cluster.
Capacity UsedThe HDFS storage used on the cluster.
Capacity Used Non DFSThe storage used by data in the data node that is not included in DFS.
Total FilesThe number of files in the HDFS cluster.
Files Under ConstructionThe number of HDFS files being written.
Lock Queue LengthThe number of threads waiting for FSNameSystem lock.
Num Active ClientsThe number of clients connected to HDFS.

You can view the following charts on the HDFS Dashboard. All charts contain aggregated values of the metrics. Note: To view usage by node, click Show Individuals in the chart of your choice.

Chart NameDescription
HDFS NameNode ProcessLive tracking for active state of HDFS Namenode process, during the selected time period.
HDFS DataNode ProcessLive tracking for active state of HDFS DataNode process, during the selected time period.
HDFS DataNode State TimelineThe number of Live DataNodes, Stale DataNode, Dead DataNodes. DecomLiveDataNodes on the cluster for the selected timeline.
CPU Usage Name NodeThe rate of change of CPU utilization by a NameNode on the host, during the selected time period.
CPU Usage Data NodeThe rate of change of CPU utilization by a DataNode on the host, during the selected time period.
Physical Memory NameNodeThe average memory (RSS - resident set size) usage by NameNode process running on a host.
Physical Memory DataNodeThe average memory (RSS - resident set size) usage by DataNode process running on a host.
Heap Memory NameNodeThe amount of heap memory used/committed/maximum by the NameNode process (in MB or GB).
Heap Memory DataNodeThe amount of heap memory used/committed/maximum by the DataNode process (in MB).
Total FilesThe total number of files on the HDFS cluster..
Files Under ConstructionThe number of HDFS files that are still being written.
Lock Queue LengthThe number of threads waiting to acquire FSNameSystemLock.
Num Active ClientsThe total number of active clients holding lease in the system.
File Summary TrendThe file summary trend showing the Total Files, Files Under Construction, LockQueueLength, and numActiveClients.
Active ConnectionsThe 95th percentile value of number of active clients holding lease in the system.
NN Running ThreadsThe number of RUNNABLE threads for NameNode process.
DN Running ThreadsThe number of RUNNABLE threads for DataNode process.
NN Waiting ThreadsThe number of WAITING and TIMED_WAITING threads for the NameNode process.
DN Waiting ThreadsThe number of WAITING and TIMED_WAITING threads for the DataNode process.
Capacity RemainingThe amount of HDFS storage available on the cluster, in MB. This is calculated as Capacity Total - (Capacity Used + Capacity used non DFS).
Capacity TotalThe total HDFS storage on the cluster.
Capacity UsedThe HDFS storage used on the cluster.
Capacity Used Non DFSThe storage used by data in the data node that is not included in DFS.
HDFS Usage TrendThe HDFS Usage summary trend showing CapacityTotal, CapacityUsed, and CapacityRemaining.
Block Summary TrendBlocks are the smallest units of storage on the host where data or files are broken down into chunks and stored in continuous manner. HDFS distributes these blocks across the Hadoop cluster. The block summary trend shows the following:
Blocks Total: Total number of blocks in the cluster.
Missing Blocks: Number of blocks having no replicas in the Hadoop cluster.
Pending Replication Blocks: Number of blocks that are not yet replicated.
Under Replicated Blocks: Number of blocks having replication factor less than the specified value.
Corrupt Blocks - number of blocks with corrupt replicas.
Data SkewnessThe Total Capacity and Capacity Used for each DataNode.
Capacity Used Non DFSThe storage used by data in the data node that is not included in DFS.
Top OperationsThe most number of HDFS file operation commands running on the HDFS.
Top User by OperationsThe users running the most number of HDFS operations.
RPC TimesThe time taken to complete RPC calls for the following criteria.
RPC Processing Time: Time taken to process RPC calls.
RPC Queue Time: Time taken for an RPC call to start, or the waiting time of an RPC call.
RPC OperationsThe number of RPC operations in the Hadoop cluster for the following types.
Processed Ops: The number of processed RPC operations.
Queued Ops: The number of RPC operations in queue and yet to start processing.
HDFS Usage by UserThe amount of HDFS storage used by each user in the cluster.