Problem

When you reach the upper limit of allowed threads while running a Java application you encounter an error like this:
java.lang.OutOfMemoryError: unable to create new native thread

Example of an error:

=========

java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.axway.clusterold.startup.Boot$MainRunnable.run(Boot.java:109)
        at java.lang.Thread.run(Unknown Source)
        at com.axway.cluster.extensions.thread.EventedThread.primRun(EventedThread.java:102)
        at com.axway.cluster.extensions.thread.EventedThread.run(EventedThread.java:80)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Unknown Source)
        at com.cyclonecommerce.util.task.TaskScheduler.startThreads(TaskScheduler.java:248)
        at com.cyclonecommerce.util.task.TaskScheduler.initialize(TaskScheduler.java:84)
        at com.cyclonecommerce.util.task.TaskScheduler.<init>(TaskScheduler.java:315)
        at com.cyclonecommerce.util.task.TaskScheduler.getInstance(TaskScheduler.java:53)
        at com.axway.b2bi.mapproxy.deployment.MapProxyCommandLineDeployer.init(MapProxyCommandLineDeployer.java:938)
        at com.axway.b2bi.mapproxy.deployment.MapProxyCommandLineDeployer.main(MapProxyCommandLineDeployer.java:544)
        ... 8 more

=========

Don't be fooled by the OutOfMemoryError, worry about what's after the colon - "unable to create new native thread". What this basically means is that the JVM is unable to create a native/OS level thread.

This is the error thrown by the JVM:

If you would like to read more about this, look here:

http://javaeesupportpatterns.blogspot.ro/2012/09/outofmemoryerror-unable-to-create-new.html

To get an idea of the threads for your user you can use the below commands:
1. To get a count of all the active thread in the system: ps -eLf | wc –l
2. To see the max allowed threads run this command: cat /proc/sys/kernel/threads-max

If the two numbers are very close together then it's time to take "charge" of your threads.

Important: This could affecting all java applications not just B2Bi/Interchange

Resolution

To solve this issue is very simple. Increase the max allowed threads.

This can be done two ways, increase the max allowed process for a specific user, or increase the global max allowed threads. Both method require root access.

On Redhat 6.5 (mostly like on many other Linux versions/distros and kernel versions) the max_threads=max_process x 2.

As root has (close to) unlimited allowed processes (in fact for root the max number of processes depends on the amount of physical memory) and regular users have 1024 (ulimit -u). As a regular user you'll quickly run out of allowed threads if your application is using a lot of threads. For a regular user you're allowed only 2048.

Below are the two methods that can be used to increase the limit:
1. change the limits.conf file (on my RHL 6.5 it's in /etc/security/limits.d/90-nproc.conf). Exert form the file:

===============
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited
===============

Either add your specific user there and increase the thread count number (higher than 1024) or change it for all the users (*).

For example - adding the line:
bobby soft nproc 4096

Will allow the user bobby to create a total of 4096 processes and with a max number of 8192 threads (max_threads=max_process x 2).

The changes to this file are permanent and will persist after a restart. Also in order for the changes to take affect the affected users need to logout and login again.

2. change directly the number of allowed threads.

Under the specific user run "cat /proc/sys/kernel/threads-max" to see how many threads you are allowed to create.

As root run echo 10000 > cat /proc/sys/kernel/threads-max to change that number.

This method affects all the users in the system, including the root.

Depending on your company policies and/or your needs choose which ever method is best suited for your project.