Preface

In the last couple of months Axway observed a spike in the cases, where the SecureTransport application started malfunctioning to varying degrees after an update of the operating system's kernel. The problem is limited to systems runing with RHEL and SLES, but will possibly affect Oracle Linux and CentOS.

Symptoms

The following details have been gathered after Axway's research.

• The issue starts after applying security patch(es) for the OS kernel
• The systems are running kernels 2.6.x, 3.x, 4,x or 5.x
• The visible effects in ST are (but are not limited to)
  
  • Enterprise Cluster nodes disconnecting and/or not joining the cluster
  • FTP, SSH and HTTP transfers are failing or timing out

What has caused it?

The exact issue was caused by the fix of CVE-2019-11478: tcp: tcp_fragment() should apply sane memory limits. This fix caused an unwanted regression - when the application creates a socket with some default value for SK_SNDBUF (socket send buffer size) and tries to put a chunk of data which is bigger than SK_SNDBUF in this socket, the kernel was dropping the data deciding this was a DoS attack attempt.

In ST's case this chunk of data was the trusted CA bundle in the FTP or HTTP SSL handshakes. The application (ST) was not notified by the kernel and from the application's point of view the data was sent. However, as the data was not sent in reality (dropped by the kernel), and ST didn't receive any kind of response from the other party, this resulted in timeouts being recorded (if configured) for the operations.

The fix for CVE-2019-11478 was the same for kernels 3.x, 4,x and 5.x and was slightly different for 2.6.x. The regression it caused was fixed a couple of weeks later (see details here) but because it was not considered critical, it was included in the regular updates released by large Linux vendors. Unfortunately it is not fixed in kernel 2.6.x which is the kernel distributed with RHEL6.x.

There is also an article published by RedHat which lists the affected kernels and provides several workarounds.

What is affected in SecureTransport?

Currently we know that ST's FTP service (kernel 3.x and later), HTTP service (on kernel 2.6.x) and the Large Enterprise Clustering's Coherence element (on kernel 3.x possibly) are affected. As of time of this writing, we are not aware of any other affected components, but it is highly possible there are such.

Remediation and recommendations

If customers are are running Linux with kernel 3.x or later they should apply the latest kernel updates.

If customers are unable to update the kernel immediately, but still can modify the system's network settings they should apply the workaround proposed by RedHat:

Disabling TCP Segmentation Offloading (TSO) and Generic Segmentation Offloading (GSO) for the transmitting NIC should avoid the issue but may affect throughput:

ethtool -K <interface> tso off gso off

If customers are running with kernel 2.6.x they should use the above workaround provided by RedHat or another workaround: set mtu_probbing = 2 as shown below:

1. Check that the default value is 0 with the command:

sysctl -a | grep tcp_mtu_probing

2. Set the new value to 2 with the command:

sysctl -w net.ipv4.tcp_mtu_probing=2

3. To make the change permanent edit file /etc/sysctl.conf and add the following line:

net.ipv4.tcp_mtu_probing = 2

Why are Axway not planning to release a fix for the ST application for kernels 2.6.x?

As the issue was caused by an OS-level component it won't be practical to "fix" it on application level. This would entail a large amount of configuration options being implemented, which will increase the complexity of the system and put an additional strain on the application's administrators to configure and manage them. The benefits of such an approach would be questionable at best, especially when a practical workaround and an official fix for later kernel versions was released by the OS vendors.

List of affected kernels

RHEL

Click on the Errata links for more details about each item and for info about the fixed packages.

Oracle Linux

Oracle Errata with the fix

SLES

SuSE 12 SP1 announcement

SLES 12 SP 4 and Appliance Platform 7.2 seems not to be affected.