Problem

SSL error RSA_public_decrypt error, TLS Alert 51

Transfer CFT version 3.0.1 SP10

After the partner renewed their certificates, the TLS handshake fails.

Trace extract:

CFTY02Z >> CTX=25000d cftpki() _ PKII PHASE=CRYPDATA CR=1 REASON=51
CFTY02Z >> CTX=25000d DECRYPT ERROR : RSA_public_decrypt() RSA_PKCS1_PADDING
CFTY18E CTX=25000d INTERNAL PKI ERROR PHASE=CRYPDATA REASON=51
CFTY18E CTX=25000d DECRYPT ERROR : RSA_public_decrypt() RSA_PKCS1_PADDING
CFTY02Z >> CTX=25000d SSLact() _ SENDING 7 ALERT DATA
CFTY02Z >> CTX=25000d 15030100 020233 >......3<
CFTY02Z >> CTX=25000d cftpki() _ PHASE=ENDSESSION
CFTY02Z >> CTX=25000d cftpki() _ PKI COMMUNICATION STRUCTURE IS

Resolution

In latest Transfer CFT versions, RSA_public_decrypt has been remplaced by RSA_verify as recommanded by OpenSSL (When generating or verifying PKCS #1 signatures, (1) RSA_sign and (2) RSA_verify should be used.)

The new functions support PKCS #1 v2.0 whereas the old ones support only PKCS #1 v1.5 padding.

New function: https://www.openssl.org/docs/man1.0.2/crypto/RSA_v...

Old function: https://www.openssl.org/docs/man1.1.0/crypto/RSA_p...

So it is not a bug, but a limitation in older Transfer CFT versions.

The version 3.0.1 is quite old now and doesn't support PKCS #1 v2.0 and TLS 1.2 that is mandatory now for all security entities.

We recommand to upgrade to a newer CFT version 3.3.2 or 3.2.4.

If it is not possible to upgrade, as a workaround, you can still use a PKCS #1 v1.5 certificate.

Ref.:

1. https://www.openssl.org/docs/man1.1.0/crypto/RSA_s...
2. https://www.openssl.org/docs/man1.1.0/crypto/RSA_v...