Problem

In some specific cases it might be beneficial to collect heap and thread dumps of the services in SecureTransport, which can be used to troubleshoot complex issues. This article will provide a guideline how to do this.

IMPORTANT! Taking heap and thread dumps is not intended as a monitoring or a health-check procedure. It must not be used on regular basis and must be done only if a complex issue is observed and requested by Axway Support or Axway RnD.

Resolution

1. Check the version of JRE that ST is using

The tools needed to execute the dumps are dependent on the type and version of Java that your ST is running. To find this out, follow these steps:

Go to $FILEDRIVEHOME/jre/bin.

Run:

./java --version

The dot-slash in ./java is IMPORTANT! Do not use java only.

You will get something like this:

openjdk 11.0.1 2018-10-16
OpenJDK Runtime Environment 18.9 (build 11.0.1+13)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

2. Get the proper JDK

You need to get the appropriate JDK for your ST. There are various sources, and one of the good ones is shown in the steps below.

Go to Adoptium's Archive.

Find and download the needed JDK version.

3. Deploy the JDK

Upload the JDK archive to the server and place it in a dedicated folder.

Extract the archive. It creates its own new sub-folder, i.e. jdk-11.0.1.

4. Prepare to run the dump commands

The below Linux terminal one-liner can be used to create both the heap and thread dumps for the TM and the HTTP service. Keep in mind the following points:

You must edit the path to the ST's run folder, on the line beginning with strun.

You must edit the path to the JDK's bin folder, on the line beginning with jhome.

You must edit the path to the folder where the dump files will be stored, on the line beginning with dumpfol.

The line beginning with httppid is getting the PID for the HTTP service. The line with tmpid does the same for the TM service. If you are to get the dumps for another service rather than HTTP (TM is always required), then you must edit the command. For example, for SSH you can use

sshpid=`cat $strun/sshd.pid`; \

and then

"$jhome"/jmap -dump:,format=b,file="$dumpfol/ssh_heap_dump_$dt.hprof" "$sshpid"; \
"$jhome"/jstack "$sshpid" > "$dumpfol/ssh_thread_dump_$dt.out"; \

instead of the HTTP-related lines.

In case the Admin service is needed, the PID file is $FILEDRIVEHOME/var/run/admin/tomcat.pid.

The jmap tools does the heap dumps, while jstack does the thread dumps.

For best results it's good to dump the data for about 5 minutes in several runs, in order to be able to compare the states. The number of runs is controlled in the for r in {1..10}; do line, and the sleep period is defined in seconds on the sleep 30; line. With these two you can adjust the command appropriately, depending on what was requested by Support or RnD.

This is the terminal one-liner:

for r in {1..10}; do \ 
    dt=`date +%Y%m%d\_%H%M%S`; \
    strun=/opt/Axway/SecureTransport/var/run; \
    jhome=/home/axway/jdk-11.0.1/bin; \
    dumpfol=/data/logs/heapdumps; if [ ! -d "$dumpfol" ]; then mkdir -p "$dumpfol"; fi; \
    httppid=`cat $strun/httpd.pid`; \
    tmpid=`cat $strun/tm-java.pid`; \
    "$jhome"/jmap -dump:,format=b,file="$dumpfol/tm_heap_dump_$dt.hprof" "$tmpid"; \
    "$jhome"/jmap -dump:,format=b,file="$dumpfol/http_heap_dump_$dt.hprof" "$httppid"; \
    "$jhome"/jstack "$tmpid" > "$dumpfol/tm_thread_dump_$dt.out"; \
    "$jhome"/jstack "$httppid" > "$dumpfol/http_thread_dump_$dt.out"; \
    sleep 30; \
done

WARNING! Each heap dump will effectively write the complete heap contents into a file. Depending on the heap size you've set for the service in your ST, the resulting log files can be very large and consume a lot of disk space. Plan accordingly when assigning the destination log folder and make sure it is on a sufficiently large disk volume. Also, check the STStartScriptsConfig file to find out what heap dump sizes you can expect.

5. Run the dump commands

Depending on what Support or RnD had requested, execute the command at the appropriate time and/or ST state.

6. Resulting log files

You should get the dump files in the destination folder, for example:

-rw------- 1 axway axway 220833143 Jul 19 17:12 http_heap_dump_20220719_171233.hprof
-rw------- 1 axway axway 220902383 Jul 19 17:13 http_heap_dump_20220719_171309.hprof
-rw-rw-r-- 1 axway axway    108913 Jul 19 17:12 http_thread_dump_20220719_171233.out
-rw-rw-r-- 1 axway axway    108913 Jul 19 17:13 http_thread_dump_20220719_171309.out
-rw------- 1 axway axway 341815978 Jul 19 17:12 tm_heap_dump_20220719_171233.hprof
-rw------- 1 axway axway 364744761 Jul 19 17:13 tm_heap_dump_20220719_171309.hprof
-rw-rw-r-- 1 axway axway    121419 Jul 19 17:12 tm_thread_dump_20220719_171233.out
-rw-rw-r-- 1 axway axway    121419 Jul 19 17:13 tm_thread_dump_20220719_171309.out

The files can be archived to a smaller file (i.e. .tar.gz file) and then provided to Support or RnD.