Problem

An SSL connection using a DH cipher, such as EDH-RSA-DES-CBC3-SHA or DHE-RSA-AES128-SHA fails with an error like:

ERROR 01/Oct/2016:00:00:00.000 [SSL alert write 0x228, 0x1150]: handshake failure [fatal].

This is especially true if the connection only starts failing after installing one of the recent OpenSSL patches or SPs.

Resolution

The most recent versions of OpenSSL have updated such that they will reject connections to servers using weak DH parameters, that is, servers vulnerable to logjam (CVE-2015-4000) with an error like the one above. This is not a bug, but a deliberate security measure, which you can read about here.

For a temporary work-around, you can force the gateway to use non-DH ciphers (those that do not begin with DHE- or EDH-), but any server with this problem has a serious security vulnerability and should be updated. Once the server is updated to use secure DH parameters, this will not be an issue.

Finally, note that there may be other reasons for a handshake failure than this one. This particular issue can only happen with DH ciphers using a weak DH parameter. In general, a packet capture and a DEBUG trace is required to troubleshoot SSL handshake errors.