KB Article #170987

SecureTransport Root Cause Analysis (RCA) - Complete Linux Checklist

Purpose of this document

The sole idea of this document is to provide information of what log files, configuration information and additional details are required when performing Root Cause Analysis regarding issues experienced with Axway's SecureTransport application. You will find generic and advanced instructions on how to collect all logging data from your SecureTransport server and provide it for review and analysis to Axway Global Support. You will not find instructions on troubleshooting or developing any custom agents or scripts which fall out of Axway Global Support's scope, thus you will be advised to contact your account representative for further assistance.


Table of contents


Important note before you send any data to Axway Global Support

Make sure to check with your company's IT Security and Legal departments before sending *any* data to Axway Global Support. All data you are about to provide to Axway must comply with your company's security policies. If necessary, feel free to use masking technique for any sensitive data (substitution, shuffling, masking out of IP addresses, login names, password hashes, etc).


Prerequisites

All lines listed below represent actual commands as executed via command line (i.e. you can use copy/paste in order to follow the steps).


Source the /etc/fd/env.sh or /opt/Axway/Securetransport/profile.sh (or the *.csh* version the scripts, depending on what Shell you are using). Examples:


source /etc/fd/env.sh*


or


source /opt/Axway/Securetransport/profile.sh*


Create a temporary folder where all logs will be stored


mkdir /tmp/`hostname`


Details About the Environment

Linux OS distribution and version (from both Edge and Backend servers)


cat /etc/*release* > /tmp/`hostname`/os_release_$(date '+%Y-%m-%d')


SecureTransport type and version: execute 'display.sh' script from Axway's Installer folder (usually located under the /opt/Axway/ folder)


$FILEDRIVEHOME/../display.sh -n securetransport > /tmp/`hostname`/st_version_$(date '+%Y-%m-%d')


A listing of patch installation history


cat $FILEDRIVEHOME/synInstall/synPatch/component.properties > /tmp/`hostname`/st_comp_prop_$(date '+%Y-%m-%d')


Cluster/streaming or standalone configuration - make sure to list all SecureTransport servers that you have in your environment (Edge or Backend servers) and what type (if any) cluster modes you are running (Standard or Large Enterprise cluster, type of database and cluster mode - i.e. active/active or active/passive).


Diagram of the environment (if available) - this will help us get familiar with the environment and better understand the current setup of the ST Edge and Backend servers and any Firewalls or Load Balancers involved within the network topology.


Detailed Description of the Issue(s)

Have you performed any recent configuration or environment changes?


Is this a performance or functionality type of issue?


How many users are/were affected? For a single account issue - provide the account's Transfer Site and Subscriptions configuration (screenshots) along with file flow description (protocol in use, server or client initiated transfer, pulls or push).


What protocol(s) has been affected/used?


Does the issue affect the core ST functionality (out-of-the-box) or additional (custom) modules or add-ons? Share any additional information in case you have developed any custom solutions/accelerators that do not come with the out-of-the-box SecureTransport installation.


In case you are able to reproduce the issue - provide detailed step-by-step instructions on how to reproduce the behavior.


OS Logs

The information bellow should be gathered from both Edge and Backend servers (if ran in cluster - from all cluster nodes (Edge and/or Backend nodes)) from the time of the occurrence of issue.


The /etc/fstab file - holds information for all devices and shares (mounts).


cat /etc/fstab > /tmp/`hostname`/fstab_$(date '+%Y-%m-%d').log


The output of df - shows information about the disk usage.


df -h > /tmp/`hostname`/df_$(date '+%Y-%m-%d').log


The /var/log/ folder (include system messages and boot information).


tar -zcvf "/tmp/`hostname`/os_logs_$(date '+%Y-%m-%d').tar.gz" /var/log


The output of dmesg - holds kernel's message buffer log.


dmesg > /tmp/`hostname`/dmesg_$(date '+%Y-%m-%d').log


The output of the ulimit command - provides information about User limits and the use of system resources.


ulimit -a > /tmp/`hostname`/ulimit_$(date '+%Y-%m-%d').log


The output of the netstat –nap command during the occurrence of the issue - provides current network statistics for the system.


netstat -anp > /tmp/`hostname`/netstat_$(date '+%Y-%m-%d').txt


The output of ps –ef during the occurrence of the issue - provides complete list of currently active processes.


ps -ef > /tmp/`hostname`/processes_$(date '+%Y-%m-%d').list


The output of crontab -l - provides a list of jobs scheduled in CRON.


crontab -l > /tmp/`hostname`/crontab_$(date '+%Y-%m-%d').log


The output of lsof – provides a list of all open files.


lsof > /tmp/`hostname`/lsof_$(date '+%Y-%m-%d').log


TCP capture from both transfer ends (i.e sending and receiving sides) - in case of network/firewall suspicion/issues.


More information: KB 151341


SecureTransport Configuration Files and Logs

The information bellow should be gathered from both Edge and Backend servers (if ran in cluster - from all cluster nodes (Edge and/or Backend nodes)) from the time of the occurrence of issue.


Current logs for the day.


tar -zcvf "/tmp/`hostname`/st_logs_folder_$(date '+%Y-%m-%d').tar.gz" $FILEDRIVEHOME/var/logs


Additional tomcat/admin logs.


tar -zcvf "/tmp/`hostname`/st_tomcat_logs_$(date '+%Y-%m-%d').tar.gz" $FILEDRIVEHOME/tomcat/admin/logs


SecureTransport application configuration files (especially the configuration.xml file).


tar -zcvf "/tmp/`hostname`/st_conf_folder_$(date '+%Y-%m-%d').tar.gz" $FILEDRIVEHOME/conf


All start scripts used tstart the protocol and TM daemons and any Java core dumps (if created)


tar -zcvf "/tmp/`hostname`/st_bin_folder_$(date '+%Y-%m-%d').tar.gz" $FILEDRIVEHOME/bin


All TM rules


tar -zcvf "/tmp/`hostname`/st_brules_folder_$(date '+%Y-%m-%d').tar.gz" $FILEDRIVEHOME/brules/local/wptdocuments/


Export of File Tracking


Use the Admin UI → File Tracking to export the transfer details for any failed transfers for 1 hour, 1 day or more (depending on when the issue happened you can also select specific time range).


Export of Server Log


Use the Admin UI → Server Log to export all logs from today or from a previous day (depending on how long the log files are kept in the database


Check the Аdmin GUI → Applications → LogEntry Maintenance application → Delete log entries when value. Log records older than this value might have been exported into files on the disk, located in $FILEDRIVEHOME/var/db/hist/logs (default location for ST instances with embedded database).


Export of Account(s)


Use the Admin UI → Accounts → Import/Export functionality to provide the complete setup of particular account(s).


Export of Server Configuration


Use the Admin UI → Server Configuration → Import/Export Server Configuration functionality to provide the complete settings of the ST server.


SecureTransport DEBUG Logging

Collecting debug logs should be performed only if necessary and in case the provided logs (with default log level) are not sufficient to identify the root cause of the issue. Keep in mind that when running SecureTransport with default log levels the log files created during the incident will not have any debug messages in them. In order to collect debug information you would need to first enable debug logging and reproduce the issue or wait for it to reoccur.


CAUTION! Running debug log level (especially for the TM) at all times is not recommended as the output of the TM log could grow very large in size and cause the database to consume a lot of disk space. Therefore, the instruction bellow also include options for exporting the logs out of the database (Admin UI → Server Log) and to be stored into flat files, i.e. text files within the $FILEDRIVEHOME/var/logs folder.


More information:


Redirecting logs to flat files Admin Guide redirect Log4j


Increase logging to DEBUG with the help of KB 182269


Collecting Java heap and thread dumps

In some specific cases it might be beneficial to collect heap and thread dumps of the services in SecureTransport, which can be used to troubleshoot complex issues.


More information: KB 182259


Rare Scenarios

In some rare occasions, the provided information and log files are not sufficient for us to determine the root cause of the issue, therefore we would ask (if possible) for a complete backup of the whole environment. This way we would be able to restore the environment, reproduce and investigate the issue internally in our lab. For the purpose, we would ask for a complete backup of $FILEDRIVEHOME.


Stop all SecureTransport services


$FILEDRIVEHOME/bin/stop_all


Archive the necessary SecureTransport instance


tar -pczf "/tmp/`hostname`/st_full_backup_$(date '+%Y-%m-%d').tar.gz" $FILEDRIVEHOME/ /etc/fd/ --exclude=$FILEDRIVEHOME/var/db/hist/*


Provide the /tmp/`hostname`/st_full_backup_*.tar.gz file


Archive and send the information to Axway

Archive all collected information in a manner that the logs are separated and organized into different packages for each server(node), by folders and sub-folders, i.e. – caseXXXXX_primary_edge_hostname-DateStamp.tar.gz or caseXXXXX_secondary_backend_hostname-DateStamp.tar.gz. Preferable compression methods or containers – use tar and gzip (or Zip).


Whenever you open a support request (case) with Axway Global Support – you will be provided with a case number (XXXXXX) and a related support account that will be used for collecting all the data throughout the investigation.


Example account information:


Server: https://st-support-emea.axway.com or https://st-support-us.axway.com (depending on the region)
Username: XXXXXXXX
Password: XXXXXXX


You can also use a FTP/SFTP client (example: SecureClient, STClient or FileZilla) to upload the files to us (FTP/s TCP21 and SFTP TCP22).

Still need help?

If this information wasn't helpful to you, just drop us a line. We'll get back to you as soon as possible.