Problem

In some specific cases it might be beneficial to collect debug logs for services in SecureTransport, which can be used to troubleshoot complex issues. This article will provide a guideline how to do this efficiently.

Caution! Before proceeding with the steps described here, it is recommended that you first redirect the logs to file. For more information refer to the Admin Guide - redirecting Log4j to files

IMPORTANT! Increasing log levels to debug is not intended as a monitoring or a health-check procedure. It must not be used on regular basis and must be done only if a complex issue is observed and requested by Axway Support or Axway RnD.

Resolution

We will go through a couple of example scenarios where one might need to get more information - debugging a server-initiated issue and a client-initiated one. For the sake of this article and the screenshots in it, the logs will not be redirected to files.

Debugging a server-initiated issue

In this example scenario, ST reached out to a remote server and tries to pull a file. The file is present but there is no transfer occurring from ST's perspective and there is no inbound transfer in File Tracking.

A server-initiated pull resulted in a WARN and ERROR messages

The following classes are showing the WARN and ERROR messages

Custom loggers created to increase logging to debug

After identifying the classes that show the messages of interest, custom loggers need to be created in the tm-log4j.xml file located in the $FILEDRIVEHOME/conf folder.

Make a backup copy of the file before editing it!

The classes we want loggers for are the ones below:

com.tumbleweed.st.server.tm.agents.PermissionsCheckAgent
com.valicert.brules.executionengine.engine.AgentJob

The rule of thumb is that one should try to create debug loggers for the right-most class, however not all classes do contain debug loggers. In other words com.tumbleweed.st.server.tm.agents.PermissionsCheckAgent on debug might not show anything useful, but com.tumbleweed.st.server.tm.agents would. The same for goes for com.valicert.brules.executionengine.engine.AgentJob and com.valicert.brules.executionengine.engine.

Furthermore, if you increase logs to debug on low lever classes, such as com.tumbleweed or com.axway - there might be too much unnecessary debug data in the logs that will make troubleshooting the issues harder, not easier.

<!-- Custom debug loggers -->
<Logger name="com.tumbleweed.st.server.tm.agents" level="DEBUG" additivity="false">
    <AppenderRef ref="ServerLog"/>
    <!--<AppenderRef ref="Stdout" />-->
    <!--<AppenderRef ref="Stderr" />-->
</Logger>
<Logger name="com.valicert.brules.executionengine" level="DEBUG" additivity="false">
    <AppenderRef ref="ServerLog"/>
    <!--<AppenderRef ref="Stdout" />-->
    <!--<AppenderRef ref="Stderr" />-->
</Logger>

New loggers should be placed at the bottom of the file, right above the last two closing tags - </Loggers> and </Configuration>

Debug log messages now present for the classes we are interested in

After saving the tm-log4j.xml file, the changes will automatically take effect in a couple of minutes. Trying to perform the same transfer now results in this output:

Identify cause for the issue

In this case the issue was caused by incorrect OS permissions of the /home/test/ar folder. The debug logs showed that we need permissions of UID/GID 12345/12345, however the actual folder was owned by a different UID/GID combination, therefore access was denied.

Debugging a client-initiated issue

In this example scenario, a client tries to login over HTTP but the login is denied. This time the logs show errors from both the HTTPD and TM components.

The logs show two errors of interest related to the client action

The following classes are showing the ERROR messages, note the component is different this time

Custom loggers created to increase logging to debug

After identifying the classes that show the messages of interest, custom loggers need to be created in the tm-log4j.xml and httpd-log4j.xml files located in the $FILEDRIVEHOME/conf folder.

Make a backup copy of the files before editing them!

The classes we want loggers for are the ones below, one per component:

TM Component, Class: com.tumbleweed.st.server.tm.agents.AccountInitializationAgent
HTTPD Component, Class: com.axway.st.server.httpd.login.LoginAuthenticator

The rule of thumb is that one should try to create debug loggers for the right-most class, however not all classes do contain debug loggers. In other words com.tumbleweed.st.server.tm.agents.AccountInitializationAgent on debug might not show anything useful, but com.tumbleweed.st.server.tm.agents would. The same for goes for com.axway.st.server.httpd.login.LoginAuthenticator and com.axway.st.server.httpd.login.

Furthermore, if you increase logs to debug on low lever classes, such as com.tumbleweed or com.axway - there might be too much unnecessary debug data in the logs that will make troubleshooting the issues harder, not easier.

With all of the above in mind, we create one custom logger per respective log4j file. In the tm-log4j.xml:

<!-- Custom TM debug loggers -->
<Logger name="com.tumbleweed.st.server.tm.agents" level="DEBUG" additivity="false">
    <AppenderRef ref="ServerLog"/>
    <!--<AppenderRef ref="Stdout" />-->
    <!--<AppenderRef ref="Stderr" />-->
</Logger>

Now in the httpd-log4j.xml:

<!-- Custom HTTPD debug loggers -->
<Logger name="com.axway.st.server.httpd.login" level="DEBUG" additivity="false">
    <AppenderRef ref="ServerLog"/>
    <!--<AppenderRef ref="Stdout" />-->
    <!--<AppenderRef ref="Stderr" />-->
</Logger>

New loggers should be placed at the bottom of the file, right above the last two closing tags - </Loggers> and </Configuration>

Debug log messages now present for the classes we are interested in

After saving the tm-log4j.xml file, the changes will automatically take effect in a couple of minutes, however the changes in httpd-log4j.xml require a restart of the HTTP daemon. Now we get the following output with debug messages for both TM and HTTPD components:

Keep in mind that when troubleshooting client-initiated issues, the changes to protocol daemon log4j file might have to be done on Edge servers. This is because the client in question can be connecting to an Edge server, thus the protocol related logs will be there.