Sometimes, Kubernetes worker nodes need node-level CPU, memory, and I/O performance data simultaneously. (For example, to diagnose SAS^® Viya^® workload bottlenecks or to provide performance evidence during a support case.) However, the standard kubectl top command does not provide the historical detail or granularity that nmon offers.

This SAS KB article applies to the SAS^® Viya^® platform (any release) running on any Kubernetes distribution, including OpenShift, Azure Kubernetes Service (AKS), and Elastic Kubernetes Service (EKS). The same approach applies to any Kubernetes workload—not only SAS Viya.

Cause

You must run nmon as a process on each node's host operating system rather than inside an isolated container. A Kubernetes DaemonSet is the correct primitive for running one pod per node simultaneously. To reach node-level resources, the pod must run with hostNetwork and hostPID enabled, and it requires a toleration for NoSchedule taints so that it can also be scheduled on control-plane or otherwise tainted nodes.

Deployment and Usage Steps

1. Deploy the nmon DaemonSet

Run the following command in a terminal with kubectl access to create the DaemonSet:

kubectl create -f - <<EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nmon-collector
  namespace: default
spec:
  selector:
    matchLabels:
      name: nmon-collector
  template:
    metadata:
      labels:
        name: nmon-collector
    spec:
      hostNetwork: true
      hostPID: true
      tolerations:
      - operator: Exists
        effect: NoSchedule
      containers:
      - name: nmon
        image: ubuntu:20.04
        command: ["bash", "-c"]
        args:
        - |
          set -e
          echo "Starting nmon setup on node: $(hostname)"
          apt-get update -qq
          apt-get install -y wget nmon
          echo "nmon installed successfully"
          mkdir -p /nmon-data
          cd /nmon-data
          echo "Starting nmon data collection..."
          nmon -f -s 30 -c 120 &
          NMON_PID=$!
          echo "nmon started with PID: $NMON_PID"
          while kill -0 $NMON_PID 2>/dev/null; do
            echo "nmon still running at $(date)"
            ls -la /nmon-data/*.nmon 2>/dev/null || echo "No nmon files created yet"
            sleep 300
          done
          echo "nmon process completed"
          echo "Final nmon files:"
          ls -la /nmon-data/
        volumeMounts:
        - name: nmon-data
          mountPath: /nmon-data
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "100m"
      volumes:
      - name: nmon-data
        hostPath:
          path: /tmp/nmon-data
          type: DirectoryOrCreate
EOF

This command creates one nmon-collector pod on every node in the cluster. Each pod installs nmon at runtime and captures 120 performance snapshots at 30-second intervals (one hour of data by default).

2. Retrieve logs and data from a specific node.

I. Find the pod running on the node that you want to inspect:

NODE_POD=$(kubectl get pods -l name=nmon-collector -o wide | grep <node-name> | awk '{print $1}')

II. View that pod's logs:

kubectl logs $NODE_POD

III. Check for errors across all nmon pods at once:

kubectl logs -l name=nmon-collector | grep -i "error\|command not found\|permission denied\|failed"

IV. Copy the .nmon output file off the node before proceeding to cleanup. The file is written to the pod's mounted hostPath volume so that you can copy it out directly: 

kubectl cp <nmon-collector-pod>:/nmon-data/<hostname>_<date>.nmon ./<hostname>.nmon

Repeat this step for each node that you need data from. Do not proceed to step 3 (Clean up) until you retrieve this file. Deleting the DaemonSet stops data collection but does not remove existing files from the node, although they should still be considered at risk of cleanup by the node's own housekeeping (for example, tmpwatch/systemd-tmpfiles on /tmp). 

3. Clean up.

Once you collect and retrieve the performance data, delete the DaemonSet to stop data collection on every node:

kubectl delete daemonset nmon-collector

Additional Information

Verification

Determine whether a pod runs on every node in the cluster:

kubectl get pods -l name=nmon-collector -o wide

Check that nmon output files are created on the node:

kubectl logs <pod-name> | grep "nmon-data"

Expected Result: One nmon-collector pod per node in Running status, and .nmon files accumulate under /tmp/nmon-data on each node.

Notes

• nmon -f -s 30 -c 120 captures 120 snapshots at 30-second intervals (one hour total). Adjust -s (interval in seconds) and -c (count) as needed for a longer or shorter capture window.
• nmon output files are written to the host at /tmp/nmon-data. Collect the output files before deleting the DaemonSet, since deleting the DaemonSet stops updates but does not remove the existing files from the node.
• The ubuntu:20.04 image installs nmon at runtime via apt (Ubuntu's package manager), which requires internet access from within the pod. For air-gapped clusters, use a pre-built image with nmon already installed.
• A test pod on a single node can validate the nmon installation before deploying the full DaemonSet:

kubectl run nmon-test --image=ubuntu:20.04 --rm -it --restart=Never -- bash -c "apt-get update && apt-get install -y nmon && nmon -h"

References

• nmon for Linux: nmon and njmon | Main / HomePage

Note: Deployment and life cycle management of the nmon DaemonSet should be owned by the Kubernetes platform administrator. The operational requirements to run this specific tool—including required cluster permissions, admission control policies (for example, Pod Security Standards or OpenShift Security Context Constraints governing hostNetwork, hostPID, and hostPath), and taint/toleration rules—vary by platform and distribution. Verify these conditions and constraints against your specific target platform before deploying the DaemonSet, rather than assuming that the defaults shown in this article will apply unchanged.