How to force multiple thread dumps against suspended SAS® Viya® platform Java based pod services


This SAS KB article explains how to force multiple thread dumps against suspended SAS Viya platform Java based pod services.

  1. Locate the name of the pod that runs the Java application for which you need the thread dump. You can list pods via the following command: 
    kubectl get pods

  2. Execute a shell inside the target pod to access the pod's shell. Replace <POD_NAME> with the actual pod name.

    Here is an example:
    kubectl exec -it sas-risk-pipeline-5bb6ffb5f8-5q7cm -- /bin/bash

    Note: You might use sh rather than bash depending on the container image.

  3. Identify the Java process ID (PID).

    Inside the pod, locate the PID of the running Java application. Submit commands similar to the following:

    ps aux | grep java 

    USERPID%CPU %MEMVSZRSSTTYSTAT START TIMECOMMAND
    sas70.10.5 3249788778392?SlJan26 262:06/usr/lib/jvm/java-21-openjdk-21.0.8.0.9-1.el9.x86_64/bin/java
  4. Generate a thread dump.

    If installed, use the jstack tool to generate the thread dump every few seconds or minutes, depending on how long the process is suspended for. 

    Replace <PID> with the PID and <FILENAME> with your desired output file name. Replace <n> with an iterative, incremental number for as many thread dumps as you take:
    jstack <PID> > <FILENAME>.tdump

    Based on the above table, here is an example ps aux output: 
    jstack 7 > threaddump<n>.tdump

    If you do not have the jstack tool (that is, the tool is not installed), submit a kill -3 <PID>
    signal every few seconds or a couple of minutes, depending on how long the process is suspended, to print the thread dump to the application pod logs file.

  5. Exit the pod bash or shell session and gather the pod log. Then, upload the log to SASTSDrive for further review. Note: Include the jstack dumps if you generated the dumps with the jstack tool. 

    Here is an example:
    sudo kubectl logs sas-risk-pipeline-5bb6ffb5f8-5q7cm -n viya >> sas-risk-pipeline-5bb6ffb5f8-5q7cm.log