Problem

-- How do you increase the number of API Gateway worker threads?

Resolution

-- There's a configurable parameter for the maximum number of threads the gateway will start: "maxThreads" in the SystemSettings element in service.xml (by default 1024 on Unix, 512 elsewhere):

<SystemSettings title="API Gateway" tracelevel="INFO" serviceName="$" serviceID="$" secret="$" groupName="$" groupID="$" domainID="$" maxThreads=”1024”/>

In response to a large burst simultaneous requests, threads will be created up to that limit: after that, new incoming calls will not be handled until one of the existing ones is finished.

The actual queue of unprocessed incoming calls is managed by the OS kernel: there is a backlog parameter (even on Windows) that you can specify for an incoming interface that limits the number of TCP connections the system will accept without having the application take responsibility for them (so if all threads are busy in the gateway, this queue will start to back up to this limit until threads become available to service them)

Often, if this queue is exceeded, incoming clients will have their TCP connections reset immediately. The kernel limit can be a bit misleading, though: if you are using "syncookies" (sometimes used to mitigate possible flooding attacks), then the limit is essentially removed, and the actual limit will be more down to the level of service the clients are willing to put up with as it degrades in the face of increasing offered load. Windows also has something similar.