Problem

an EBCDIC file of size 1.2GB is picked from a file system and then sent to a partner to a mainframe over FTP
when the "Select the file type" is set to "Binary" in the Partner Delivery of type FTP (external FTP server), the transfer works fine
if the "Select the file type" is accidentally set to "Ascii - automatic line ending", a Java heap space occurs, and triggers the restart of the node

2017-03-21 18:05:29,500 - INFO [Worker42-6432] (NamedConsumerDriver.consume:119) - Consumed file /opt/axway/shared/b2bi/common/data/out2/.ready/TEST.ILA046 from /opt/axway/shared/b2bi/common/data/out2 in exchange xxx
2017-03-21 18:07:58,242 - ERROR [Worker46-6437] (WorkerThread.primRun:403) - Caught an OutOfMemoryError. Restarting
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)
at java.lang.StringBuffer.append(StringBuffer.java:369)
at java.io.BufferedReader.readLine(BufferedReader.java:370)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at com.cyclonecommerce.tradingengine.transport.ftp.commands.Stor.executeDataIo(Stor.java:124)
at com.cyclonecommerce.tradingengine.transport.ftp.commands.FtpDataCommand.handleData(FtpDataCommand.java:151)
at com.cyclonecommerce.tradingengine.transport.ftp.commands.FtpDataCommand.execute(FtpDataCommand.java:124)
at com.cyclonecommerce.tradingengine.transport.ftp.FtpClientImpl.executeCommands(FtpClientImpl.java:1737)
at com.cyclonecommerce.tradingengine.transport.ftp.FtpClientImpl.send(FtpClientImpl.java:826)
at com.cyclonecommerce.tradingengine.transport.system.production.producers.FtpProducer.send(FtpProducer.java:121)
at com.cyclonecommerce.tradingengine.transport.system.production.producers.FtpProducer.produce(FtpProducer.java:80)
at com.cyclonecommerce.tradingengine.transport.system.production.ProducerTask.produce(ProducerTask.java:280)
at com.cyclonecommerce.tradingengine.transport.system.production.ProducerTask.execute(ProducerTask.java:135)
at com.cyclonecommerce.util.task.TaskScheduler$WorkerThread.primRun(TaskScheduler.java:394)
at com.axway.cluster.extensions.thread.EventedThread.run(EventedThread.java:80)
2017-03-21 18:08:35,774 - INFO [Startup] (ClusterController.connect:63) - Node b2b-server_te, of type TE is connecting to the cluster
2017-03-21 18:08:35,946 - INFO [Startup] (AcceptorThread.bindServerSocket:86) - Listening for node connections on port 47002
2017-03-21 18:08:38,104 - INFO [Startup] (MembershipManager.addConnection:131) - Node b2b-server_te has joined the cluster
2017-03-21 18:08:38,270 - INFO [ConnectTo: b2b-server_cn] (MembershipManager.addConnection:131) - Node b2b-server_cn has joined the cluster
2017-03-21 18:08:38,301 - INFO [ConnectTo: b2b-server_b2b] (MembershipManager.addConnection:131) - Node b2b-server_b2b has joined the cluster

should the transfer fail in a better way instead of throwing memory errors / improve the error handling ?

Resolution

* B2Bi cannot control the OOM. If the system remains with no resources, than it will automatically restart. There is no way B2Bi can catch an OOM and continue the processing of other messages and just fail this one "nicer" because there is no more memory to continue.

* When the file type is set to ASCII, the entire file is loaded in memory searching for the separators that are never found

It is not the case when the file type is set to BINARY

* To avoid the OOM in case the file type has been set "accidentally" the memory needs to be increased at the OS level and / or in the jvmArguments.xml