Problem

Direct buffer memory typically occurs when the Java Virtual Machine (JVM) runs out of native memory allocated for direct buffers. Direct buffers are used by the java.nio package, which provides non-blocking I/O operations and allows Java programs to work more efficiently with native code or native memory. Direct buffer memory are not present in the heap memory, increasing or decreasing the heap size won't help.

ERROR P-11-MPX-1                             outcall : [Outcall:xxx] Outcall to SAP hostname/IP:port failed: com.axway.niocore.buffer.NIOOutOfMemoryException: java.lang.OutOfMemoryError: Direct buffer memory
com.axway.niocore.buffer.NIOOutOfMemoryException: java.lang.OutOfMemoryError: Direct buffer memory
    at com.axway.niocore.buffer.NIOBufferFactory.allocateDirect(NIOBufferFactory.java:49) ~[commons.jar:2.30.0-1]
    at com.axway.niocore.communicator.Communicator.<init>(Communicator.java:62) ~[commons.jar:2.30.0-1]
    at com.axway.niocore.communicator.ClearCommunicator.<init>(ClearCommunicator.java:33) ~[commons.jar:2.30.0-1]
    at com.axway.niocore.connector.Connector.connectEvent(Connector.java:281) [commons.jar:2.30.0-1]
    at com.axway.niocore.NIOCore.run(NIOCore.java:316) [commons.jar:2.30.0-1]
    at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.nio.Bits.reserveMemory(Unknown Source) ~[?:?]
    at java.nio.DirectByteBuffer.<init>(Unknown Source) ~[?:?]
    at java.nio.ByteBuffer.allocateDirect(Unknown Source) ~[?:?]
    at com.axway.niocore.buffer.NIOBufferFactory.allocateDirect(NIOBufferFactory.java:47) ~[commons.jar:2.30.0-1]

Resolution

1. You can try to insert "+DisableExplicitGC" option in the profile of DMZ node p_secure_relay_jvm_extra_parameters="-Xmx512m -Djava.security.debug=provider,engine=Cipher,KeyAgreement,KeyGenerator,KeyPairGenerator,KeyStore,Mac,MessageDigest,Signature"+DisableExplicitGC. This option will prevent JVM to perform the garbage collection
2. Increase the DirectBuffer memory size to 2GB: p_secure_relay_jvm_extra_parameters="-Xmx512m -XX:MaxDirectMemorySize=2G -Djava.security.debug=provider,engine=Cipher,KeyAgreement,KeyGenerator,KeyPairGenerator,KeyStore,Mac,MessageDigest,Signature"