Problem

OpenSSL before 0.9.8l, and 0.9.8m through 1.x, does not properly restrict client-initiated renegotiation within the SSL and TLS protocols, which might make it easier for remote attackers to cause a denial of service (CPU consumption) by performing many renegotiations within a single connection, a different vulnerability than CVE-2011-5094. NOTE: it can also be argued that it is the responsibility of server deployments, not a security library, to prevent or limit renegotiation when it is inappropriate within a specific environment.

Resolution

ST 5.1 running on Unix-based environments does not support client-initiated renegotiation. The renegotiation is disabled as mitigation for CVE-2009-3555. ST 5.1 running on Windows uses newer version of OpenSSL, which implements RFC 5746 (Transport Layer Security (TLS) Renegotiation Indication Extension). This means that the reported herein security vulnerability can be applicable for ST 5.1 running on Windows.

The TLS protocol is a cryptographic protocol, and servers that expect to talk to real clients performing the protocol as designed will attempt to perform the cryptographic operations as requested. Because of the processing power required to handle an SSL/TLS handshake, with renegotiation enabled, a user can send multiple handshakes per second due to the renegotiation request being permitted. This could allow a malicious user to send multiple renegotiation requests and exhaust server resources. Though, a DoS-client can simply open new connections to the SSL server and blindly fire away SSL handshake messages, forcing the server to do the same crypto work. That will in the very same way make most servers perform cryptographic operations and exhaust server resources. Having this said, the DoS attack via repeated SSL session renegotiations offers limited advantages over an attack were only one (initial) negotiation is performed over one connection. The main difference is that using renegotiation allows making this attack more efficient by using single connection for several handshakes and bypasses arbitrary limits into applications or firewalls for restricting connection rates.

However, from cryptographic point of view - once the crypto operation (herein part of the handshake) has been initiated, it has to complete at full crypto cost. So when we refer to the renegotiation problem, there currently isn't a conclusion whether this issue should be addressed in the TLS protocol and the libraries implementing it because it is inherent property of the protocol that client can "DoS" the server by making it perform expensive cryptographic operations. Therefore, the cryptographic libraries (like OpenSSL, and specifically OpenSSL) does not provide fixes for this issue.