Problem

Security Scans may report that SSL/TLS endpoint (SecureTransport) has client-initiated renegotiation enabled.

Background

SSL/TLS client-initiated renegotiation is a feature that allows the client to renegotiate new encryption parameters for an SSL/TLS connection within a single TCP connection. During the SSL/TLS handshake the server incurs a higher computational cost. An attacker can exploit this asymmetry by opening an SSL/TLS connection to the server and repeatedly initiating renegotiation, causing the server to waste resources that would otherwise be used for the server's normal function.

Traditional computational DoS attacks against servers that support TLS involve opening separate TCP connections to a server, so DoS-prevention techniques typically block a client once it has opened a large number of TCP connections to the server. Renegotiation-based computational DoS has the benefit of opening only a single TCP connection to the server, allowing the attacker to more easily evade detection by this heuristic and the attacker does not have to perform a TCP handshake with the TLS server for each TLS handshake completed. However, renegotiation-based computational DoS attack may not exhaust resources as effectively as traditional or distributed DoS attacks when the server uses a thread per connection.

Opening numerous TCP connections (i.e. creating more threads) is more likely to deny service to other users than opening a single connection (i.e. creating one thread) and repeatedly initiating renegotiations. This is because, under a fair scheduler, each thread is given an equal amount of processing time, so while the renegotiations will keep that one thread busy during its slice of processing time, the other threads will function normally. Additionally, a renegotiation-based computational DoS attack is also less efficient than a traditional computational DoS attack against TLS since the client must perform additional cryptographic computations to conduct the attack. Enabling SSL/TLS client-initiated renegotiation may allow an attacker to conduct a denial of service (DoS) attack against the server, disrupting its availability by triggering a multitude of TLS handshakes using a single TCP connection. In addition to consuming computing resources, DoS can be used to preoccupy security personnel in order to facilitate a subsequent attack.

SecureTransport and Axway's Recommendation

SSL/TLS client-initiated renegotiation is enabled by default on SecureTransport. SecureTransport 5.4 supports disabling the client renegotiation feature and it is the recommended action.

Resolution

Disabling SSL/TLS client-initated renegotiation

1. Backup the files:

$FILEDRIVEHOME/bin/start_httpd
$FILEDRIVEHOME/jre/lib/security/java.security

2. Edit the start_httpd script and add the following JAVA_OPTS line (you can add it on top of the #BEGIN GC LOGGING line):

JAVA_OPTS="-Djdk.tls.rejectClientInitiatedRenegotiation=true $JAVA_OPTS"

3. Edit the java.security file and add the following line:

jdk.tls.rejectClientInitiatedRenegotiation=true

4. Restart all STservices.

Verifying the solution

1. Download the SSL scanning tool SSLyze from GitHub

2. Using SSLyze run the following command against the HTTP service of ST:

sslyze.exe --regular SERVER_ADDRESS

where SERVER_ADDRESS is the hostname or IP of the ST server.

The expected result in the report should be:

....
* Session Renegotiation:
Client-initiated Renegotiation: OK - Rejected
Secure Renegotiation: OK - Supported