Problem

An error like transient failure connecting to remote: example.com:80: connection failed: too many open connections is thrown and causes a connection to fail, for example:

DEBUG 01/Jan/2018:00:00:00.000 run filter [Connect to URL] {
DEBUG 01/Jan/2018:00:00:00.000 Adding MessageListener: com.vordel.circuit.net.ConnectionProcessor$1@00000000
DEBUG 01/Jan/2018:00:00:00.000 get connection to host example.com port 80 scheme http
DEBUG 01/Jan/2018:00:00:00.000 Trying to acquire connection to: example.com:80 , connection count: 128, max connections: 128
DEBUG 01/Jan/2018:00:00:00.000 Failed to acquire connection to: example.com:80
ERROR 01/Jan/2018:00:00:00.000 transient failure connecting to remote: example.com:80: connection failed: too many open connections
DEBUG 01/Jan/2018:00:00:00.000 connection processor made 1 attempts to transact
DEBUG 01/Jan/2018:00:00:00.000 } = 0, filter [Connect to URL]

Resolution

The message here is that there are too many open connections for the endpoint in question. By default there is an upper limit of 128 connections. If there is already a remote host configure for this connection, then you can either increase or remove that limit

In Policy Studio and going to the Instance where the Remote connection is configured under API Gateway > Listeners > API Gateway. Select the Remote Host and go to Edit. If there is no remote host, you will need to create one for the hostname or IP you are connecting to and the port.

Under Edit you will find the General tab where you will see Maximum Connections. You can verify the number of connections used by setting the trace level to DEBUG and comparing the connection count to the max connections value. You can increase the number in the remote host, or set it to -1 to remove the limit, then deploy the config to the instance. Please note that entering -1 for Maximum Connections allows for an unlimited number of connections and that the value for the active timeout is used to decide how long to wait for a connection slot to open. So if you use a large active timeout, like the default value of 30 seconds, you will want to set Maximum Connections to -1 to avoid letting connections that can't be serviced yet build up in the gateway for a long period of time. If clients retry faster than the API Gateway gives up on waiting, that can cause a "retry storm" that makes all Gateway traffic fail until it is rebooted. If you see this error repeated many times and then the gateway stops processing any traffic, that's what is happening and you should set Maximum Connections to -1 to avoid it in the future.

This error message should not be confused with the "too many open files" errors which indicate instead that the gateway is hitting the ulimit -n value in Linux. Refer to KB 169807 for information on setting a higher ulimit value.