Problem

The API Gateway appears not to be sending the body of an HTTP message. Consider a DATA trace that shows the gateway sending a message like this:

POST /example HTTP/1.1 
Host: 10.0.0.1:8080
Via: 1.1 EXAMPLEPROXY () 
Authorization: Basic SSBrbmV3IHlvdSB3b3VsZCB0cnkgdG8gZGVjb2RlIHRoaXMgOlA=
Expect: 100-continue 
Accept: */*
Accept-Language: en-us 
Cache-Control: no-cache 
User-Agent: Mozilla 
Transfer-Encoding: chunked 
Connection: keep-alive 
X-CorrelationID: Id-000000000000000000000001 1 
Content-Type: text/xml

In the data trace there is no message body, only the headers are sent. This is later followed by a timeout. For example:

ERROR    1/1/14 00:00:00.000    nested fault: timeout. error reading from remote '10.176.98.43:8182': java.lang.RuntimeException: timeout. error reading from remote '10.0.0.1:8080'

Resolution

For HTTP GET requests, this is expected behavior as documented in KB 176710. But, in the example above we have a POST request and we can see from the 'via' header that there is an HTTP/1.1 connection being made to 'EXAMPLEPROXY' and that we have sent an Expect: 100-continue header. The fault in this example lies with the proxy. When we sent 100-continue the protocol requires the other side to reply with an HTTP status code and it did not follow the protocol. This leaves us waiting for a reply that will not come and that results in a timeout error.

If fixing the proxy is not an option, you can work around this by setting a remote host to force the connection to use HTTP/1.0, which does not use 100-continue. Note that there may be a performance impact for this. 100-continue allows you to fail early in a connection, before sending large payloads. If the example BASIC authentication shown in the sample request had been incorrect, or there was any other fault that could be detected from the headers alone, the connection would have failed when a 401 Unauthorized response was returned without ever sending the payload. When using HTTP 1.0, we have to transmit the entire message before getting a response. Note that it is also possible for the endpoint to just not respond (for example, due to hitting an idle timeout). Disabling 100-continue is necessary when the endpoint or proxy never respond to 100-continue, not just when a particular failure happened at that time.

Finally, it is also possible to disable the 100-continue mechanism globally by setting the dont.expect.100.continue property to true in jvm.xml though 100-continue will still be sent if BASIC authentication or Kerberos credentials are used via a credential profile. Refer to KB 179633 for an example of setting the property and to the documentation on system property changes for more information on how to set JVM properties in general. This is a global change, but it is generally a good one for API traffic, where credentials are not usually user-supplied and therefore the endpoint will almost never reject large messages based on the header. Disabling 100-continue gives better latency numbers for average traffic by saving the latency cost of a round trip to the back end server.