Problem

In some cases, a successful connection can be established from a Mainframe to ST 5.5 over FTPS, but when the Mainframe FTP client tries to open a passive connection (for example when listing files/folders, or when transfering files), the connection can fail with the below errors logged in the Mainframe logs:

>>> PASV
227 Entering Passive Mode (10,11,12,13,85,55)
>>> LIST
150 Opening ASCII mode data connection for file list.
protDataConnAttls: Issuing SIOCTTLSCTL to query policy state
protDataConnAttls: AT-TLS policy set as application controlled.
TTLSRule: Default_FTP~1
TTLSGroupAction: gAct1
TTLSEnvironmentAction: eAct~FTP~1
TTLSConnectionACtion: eAct~FTP~1
protDataConnAttls: reusable Peer_ID set and session reuse is allowed
protDataConnAttls: Issuing SIOCTTLSCTL to start handshake
protDataConnAttls: ioctl() failed on SIOCTTLSCTL - EDC8121I Connection reset. (errno2=0x77A9733D)
getNegotiatedTLSvalues: ioctl() failed on SIOCTTLSCTL - EDC8124I Socket not connected. (errno2=0x77B77221)
CA1582 SETCEC code = 17
TLS security mechanism negotiation failed - data connection closed
425 Can't build data connection: Connection refused.
setClientRC: entered
setClientRC: std_rc=14425, rc_type=STD, rc=14425
Std Return Code = 14425, Error Code = 00017
>>> QUIT
221 Goodbye.

On ST's side, the following error may be observed in the FTP log:

com.axway.st.server.ftpd.handler.StFtpHandler  - Exception caught, closing session
java.io.IOException: Connection reset by peer
    at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at java.base/sun.nio.ch.SocketDispatcher.read(Unknown Source)
    at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
    at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
    at java.base/sun.nio.ch.IOUtil.read(Unknown Source)
    at java.base/sun.nio.ch.SocketChannelImpl.read(Unknown Source)
    at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:280)
    at org.apache.mina.transport.socket.nio.NioProcessor.read(NioProcessor.java:1)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:695)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:668)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:657)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$9(AbstractPollingIoProcessor.java:654)
    at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1141)
    at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)

Resolution

In the Mainframe FTP client configuration (FTP.DATA configuration) check for the SECURE_SESSION_REUSE option, and if the value is ALLOWED set it to NONE and retry the transfers.