Problem

During a TLS handshake the server presents a list of the distinguished names of acceptable certificate authorities. In ST this list contains the Common Names (CN) of all certificates present in the "Trusted CA Certificates" store. This list is sent by the ST server to the clients when the option Client Certificates is set to "optional" and Accept Certificates Issued By is set to "Any Trusted Issuer".

Configuration settings under Authentication → Login Settings

Here is how the CN list looks like (this is just the beginning of the list) in a TLS session:

Acceptable client certificate CA names
/C=US/O=Equifax Secure Inc./CN=Equifax Secure Global eBusiness CA-1
/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - EC1
/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Global Root CA
/C=JP/O=SECOM Trust.net/OU=Security Communication RootCA1
...

According to RFC 2246 TLS Protocol Version 1.0 section 7.4.4 there is a size limitation of this list - its length shouldn't exceed 65535 i.e 2^16-1 bytes.

DistinguishedName certificate_authorities<3..2^16-1>;

certificate_authorities
A list of the distinguished names of acceptable certificate authorities. These distinguished names may specify a desired distinguished name for a root CA or for a subordinate CA; thus, this message can be used both to describe known roots and a desired authorization space.

The size limit could be reached with a different number of certificates, because the length of a CN varies greatly. With typical CA certificates in the trusted CAs store around 500 are needed to reach the limitation.

If the list exceeds the limit the ADMIN will become inaccessible for new connections. Java will be unable to prepare a server hello message and a TLS session to the admin port will not be possible. Here is how an attempt to connect from OpenSSL looks like:

root@rhel7-axw1:/opt/Axway/SecureTransport/bin# openssl s_client -connect 127.0.0.1:444
CONNECTED(00000003)
140639469914016:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1621630509
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

When the issue is present you may see the following message in the admin services's catalina.out log file:

Jan 21, 2022 11:55:09 PM org.apache.tomcat.util.net.NioEndpoint$SocketProcessor doRun
SEVERE: Error running socket processor
java.lang.RuntimeException: Field length overflow, the field length (66803) should be less than 65536
        at sun.security.ssl.HandshakeOutStream.checkOverflow(HandshakeOutStream.java:164)
        at sun.security.ssl.HandshakeOutStream.putInt16(HandshakeOutStream.java:108)
        at sun.security.ssl.CertificateRequest$T12CertificateRequestMessage.send(CertificateRequest.java:543)
        at sun.security.ssl.SSLHandshake$HandshakeMessage.write(SSLHandshake.java:552)
        at sun.security.ssl.CertificateRequest$T12CertificateRequestProducer.produce(CertificateRequest.java:623)
        at sun.security.ssl.SSLHandshake.produce(SSLHandshake.java:420)
        at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:964)
        at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:718)
        at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:683)
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:376)
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:983)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:970)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:917)
        at org.apache.tomcat.util.net.SecureNioChannel.tasks(SecureNioChannel.java:455)
        at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:519)
        at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:243)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1662)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

To check the current size of the CN list use the below command.

echo QUIT | openssl s_client -connect ST_SERVER_ADDRESS:ADMIN_PORT 2>/dev/null | grep ^/ | wc -c

Replace ST_SERVER_ADDRESS and ADMIN_PORT with their respective values for the ST instance which you want to check.

Resolution

To restore the access to ADMIN you must disable the client certificate authentication. This can be done by adding the following line to <FILEDRIVEHOME>/conf/options-overwrite.conf and restarting the admin service:

Admin.ClientCertificateAuthentication=none

After restoring access to the Admin UI you must sanitize the "Trusted CA Certificates" list by removing old, expired or not-in-use CA certificates, before re-enabling the client certificates authentication for administrators with the original options - Client Certificates set to "optional" and Accept Certificates Issued By set to "Any Trusted Issuer". Keep in mind to remove the line from the <FILEDRIVEHOME>/conf/options-overwrite.conf file before the next restart of the admin service.

Axway is working on a solution for this issue in ST 5.5.