Problem

Below errors appear in the trace with cluster node crashing and dumps are generated in the core folder with out of memory errors.

ERROR :20151023:17.05.48.66:r4edi(b2bi_superviser):Thread:858:array index out of bounds

ERROR :20151023:17.05.48.66:r4edi(b2bi_superviser):Thread:835:COROUTINE RESUME: non-existing coroutine ID

ERROR :20151023:17.05.48.66:r4edi(b2bi_superviser):B2Bi Supervisor:113:EXECUTE: COROUTINE RESUME: non-existing coroutine ID

ERROR:20151023:17.05.48.66:r4edi(b2bx_server):systemmonitor.s4m:263:FORCED

STOP: Reason B2Bi Supervisor process stopped for Node frgnbb2b03

WARNING:20151023:17.05.48.66:r4edi(b2bx_server):systemnode.s4m:666:FORCED STOP - detected on line 666

Resolution

The cause of the problem may be an insufficient ulimit setting, try by increasing the nproc value at the OS level. Dumps are generated with failed to create threads with out of memory error.

"nproc" limit affects the number of processes allowed by a single user running on system.