Problem

-- SSL connections, e.g. to a server (AS2, FTP, or OFTP) fail

-- A certificate presented by the server in the certificate path shows 1024 bit or lower when inspected via Windows certificate manager

-- Error message in Secure Relay router.log for incoming connection:

SocketVirtualChannel : #70#: communicator raised an exception during
HANDSHAKE, closing
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
        at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
        at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
        at com.axway.niocore.communicator.SSLCommunicator.wrap(SSLCommunicator.java:532)
        at com.axway.niocore.communicator.SSLCommunicator.writeEvent(SSLCommunicator.java:473)
        at com.axway.niocore.NIOCore.run(NIOCore.java:331)
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
        at com.axway.niocore.task.Worker.run(Worker.java:76)
Caused by: java.security.cert.CertificateException: Certificates does not conform to algorithm constraints
        at sun.security.ssl.AbstractTrustManagerWrapper.checkAlgorithmConstraints(SSLContextImpl.java:1055)
        at sun.security.ssl.AbstractTrustManagerWrapper.checkAdditionalTrust(SSLContextImpl.java:1023)
        at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:937)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)

Resolution

1. Inspect all certificates presented by the server via openssl:
   $ openssl x509 ServerCert.cer
2. Check line starting with "Public-Key:"
   Public-Key: (1023 bit)
3. Any key below 1024 bits is not allowed by Java in a certificate.
4. Windows certificate manager does not show the accurate number, but "rounds" up to 1024 (other application might do this as well)

The recommended solution is to replace the existing certificate on the server with a new one using more than 1023 bits (e.g. 2048 or 3072 bits). As a workaround, the client can allow smaller bits in the certificate by updating the $ACTISEDI/../Java/[OS]/jre[x_y_z]/lib/security/java.security file.

1. Backup and open the java.security file with a texteditor
2. Find "jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024"
3. Lower the "1024" to the same lower value the offending certificate has
4. Save changes
   
5. Restart the application (IS or AJAS)