KB Article #163027

Find High CPU Using Threads

Problem

-- What is needed for a root cause analysis prior to restarting the application if a high CPU usage is seen?

Resolution


1) Capture all Interchange logs. If logs are saved in the default location, use the diagnose tool found at [installDir]/Interchange/tools to create a zip containing the basic information (bin, conf, logs and site folder).


2) Find the java thread that causes the high CPU utilization, this differs between the OS Java is running on:


  • Find the process ID that has the high CPU utilization, via top command
  • Solaris:
    1. Issue "prstat -L -p [PID] 1 1" and capture the output
    2. PROCESS/LWPID correcsponds to Light Weight Process ID (Java thread ID)
    3. Issue "pstack" command and capture the output
    4. Generate thread dumps of all java processes of the Interchange application by using the "kill -3 [PID] command. Take three thread dumps for each process, each one minute apart
    5. Convert the Java Thread ID from decimal to Hexadecimal
    6. Search the thread dump for a corresponding nid = [Java Thread ID in HEX]
  • Linux:
    1. issue "top -Hc"
    2. This will not only show Java processes, but also all Java threads
    3. Capture the output at least three times after the top refreshed the screen
    4. Generate thread dumps of all java processes of the Interchange application by using the "kill -3 [PID] command. Take three thread dumps for each process, each one minute apart
    5. It is important that each set of screenshots and thread dumps are taken at the same time
    6. Convert the [pid] of the thread with the highest CPU utilization from decimal to hexadecimal
    7. Search the thread dump for a corresponding nid = [PID in HEX]
    • AIX:
      1. Issue "ps -mp [Java PID] -o THREAD" and capture the output
      2. Find TID of the thread with the highest CP value
      3. Generate thread dumps of all java processes of the Interchange application by using the "kill -3 [PID] command. Take three thread dumps for each process, each one minute apart
      4. Convert the TID decimal value to hexadecimal
      5. Search the thread dump for a corresponding nid = [TID in HEX]
      • Windows:
        1. Download and install Microsoft's "Process Explorer": http://technet.microsoft.com/en-us/sysinternals/bb...
        2. For Interchange 5.10.0 and later, download and install the JDK corresponding to the JRE used by Interchange (for 5.10.x that is 1.6)
        3. Use ProcessExplorer to find the java.exe with the highest CPU
        4. Right-click on the Java.exe and select "Properties"
        5. Go to the "threads" tab and order by CPU, capture the output
        6. Generate thread dumps of all java processes of the Interchange application by using the Java JDK's "jstack [pid] >> PID[pid].dmp" command. Take three thread dumps for each process, each one minute apart. Please note that if you logged in via RDP, you might need to connect using mstsc /admin or mstsc /console (depending on OS version). The jstack needs to be issued by the same user that started the Java process, the Microsoft tool PsExec might be needed to accomplish this. Downloadable at http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx.
        7. Search the thread dump for the TID captured in the ProcessExplorer
        •

        Still need help?

        If this information wasn't helpful to you, just drop us a line. We'll get back to you as soon as possible.