Problem

The error observed in the API Gateway trace is "bad record mac [fatal]" which occurs sometimes. This will be from receiving a request on a HTTPS listener interface as seen in the data trace below.

DEBUG 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] incoming call on <span class="code-keyword">interface</span> *:443 from 192.168.1.75:64575 
DEBUG 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] <span class="code-keyword">new</span> connection 0x7feb540249f0, settings source incoming <span class="code-keyword">interface</span> (allow 1.1=yes, idleTimeo 
ut=60000, activeTimeout=60000, maxConnections=128, contentLength: req=no, res=no) 
DEBUG 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] option <span class="code-quote">"nosslv2"</span> enabled <span class="code-keyword">for</span> ssl connection 
DEBUG 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] option <span class="code-quote">"nosslv3"</span> enabled <span class="code-keyword">for</span> ssl connection 
DEBUG 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] push SSL protocol on to connection 
DATA 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] [SSL_accept, 0x6000] before/accept initialization. 
DEBUG 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] No SSL host name provided, defaulting to certificate: \{ subject: /C=US/ST=My State/L=My Town/O=MyCompany/CN=My Domain }. 
DATA 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] [SSL_accept, 0x2110] SSLv3 read client hello A. 
DATA 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] [SSL_accept, 0x2130] SSLv3 write server hello A. 
DATA 25/Sep/2017:14:59:48.641 [8c81:000000000000000000000000] [SSL_accept, 0x2140] SSLv3 write certificate A \{ subject: /C=US/ST=My State/L=My Town/O=MyCompany/CN=My Domain }. 
DATA 25/Sep/2017:14:59:48.643 [8c81:000000000000000000000000] [SSL_accept, 0x2150] SSLv3 write key exchange A. 
DATA 25/Sep/2017:14:59:48.643 [8c81:000000000000000000000000] [SSL_accept, 0x2160] SSLv3 write certificate request A. 
DATA 25/Sep/2017:14:59:48.643 [8c81:000000000000000000000000] [SSL_accept, 0x2100] SSLv3 flush data. 
DATA 25/Sep/2017:14:59:48.658 [8c81:000000000000000000000000] [SSL_accept, 0x2180] SSLv3 read client certificate A. 
DATA 25/Sep/2017:14:59:48.658 [8c81:000000000000000000000000] [SSL_accept, 0x2190] SSLv3 read client key exchange A. 
DATA 25/Sep/2017:14:59:48.658 [8c81:000000000000000000000000] [SSL_accept, 0x21a0] SSLv3 read certificate verify A. 
ERROR 25/Sep/2017:14:59:48.672 [8c81:000000000000000000000000] [SSL alert write 0x214, 0x21c0]: bad record mac [fatal].* 
ERROR 25/Sep/2017:14:59:48.672 [8c81:000000000000000000000000] [SSL_accept, 0x21c0]: error - SSLv3 read finished A.* 
DEBUG 25/Sep/2017:14:59:48.672 [8c81:000000000000000000000000] destroying connection 0x7feb540249f0 with transaction (nil) 
DEBUG 25/Sep/2017:14:59:48.672 [8c81:000000000000000000000000] delete connection 0x7feb540249f0 
ERROR 25/Sep/2017:14:59:48.672 [8c81:000000000000000000000000] error handling connection: SSL protocol error 
error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac, source location: s3_pkt.c:535* 

A Wireshark/tcpdump of the connection will show the SSL handshake being terminated after an TLSv1 encrypted Alert (Level: Fatal, Description Bad Record MAC). This occurs after "Change Cipher Spec".

Resolution

There is a known problem with DH ciphers in Java 6 and Java 7 which has been fixed in

a) Java 8

b) Java 7 u6 (b06) http://bugs.java.com/view_bug.do?bug_id=2222432

c) Java 6 u115 (b32) (http://bugs.java.com/view_bug.do?bug_id=8154780)

How the bug manifests itself (description from Stack Overflow):

https://stackoverflow.com/questions/40964961/intermittent-javax-net-ssl-failure-bad-record-mac

Fix described in Java 6u115 b32:

http://www.oracle.com/technetwork/java/javase/overview-156328.html

Note that an Oracle account is required to get the update to Java 6u115:

From http://www.oracle.com/technetwork/java/javase/downloads/index.html

"Java SE 7 and Java SE 6 updates

Updates for Java SE 7 released after April 2015, and updates for Java SE 6 released after April 2013 are only available to Oracle Customers through My Oracle Support (requires support login). "

It is recommended that any Java 1.6 client is upgraded to at least Java 6 u115b32 to prevent this error.

If it is not possible to upgrade the version of Java as detailed above then the only other possibility is to remove

DH/DHE/ECDH/ECDHE ciphers in API Gateway or Java security properties for clients:

jdk.tls.disabledAlgorithms=SSLv3, DH, DHE, ECDH, ECDHE