TimeoutExceptions on REST Client - with fix proposal

[Sanne]

Elasticsearch version:

5.3

Plugins installed: N/A

JVM version: All

OS version: Any

While running integration tests for the Hibernate Search project we noticed some requests failing for TimeoutException on rather slow machines. I initially thought it was a "just a timeout", but investigating I actually identified a problem caused by the default configuration applied by the Elasticsearch REST client.

The Apache Async HTTP client being used has a pool for Non-blocking HTTP connections, implemented by org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager. This pool has a connection property which can be set via org.apache.http.client.config.RequestConfig.Builder.connectionRequestTimeout, but the name seems to have been confusing (it certainly confused me).

This configuration property validates if the Request is stale; for some reason it's also applied to instances of org.apache.http.nio.pool.LeaseRequest , and the next lease instance is validated from a queue of requests even after the execution of a successul client Request. When this second request is expired, the whole execution is marked with a TimeoutException.

To clarify: the client code sends a request A to the Elasticsearch server. A is processed fine, and the response is handled fine as well. after this, the state of A is marked as "timed out" because the next request in the PoolingNHttpClientConnectionManager is an expired lease instance.
I suspect this is a bug in the Apache Async client code as the error handling is essentially returning an exception which seems out of place.

The solution for us was extremely simple: set this property to zero, to disable this lease mechanism.

This mechanism is disabled by default in the Apache Async Http library, but the Elasticsearch client overrides this with its own default:
org.elasticsearch.client.RestClientBuilder.DEFAULT_CONNECTION_REQUEST_TIMEOUT_MILLIS

I'd recommend to set this default to zero as well. The name of the property is probably misleading? It's not really applied to the request processing on the Elasticsearch server, it's applied to the "request to the connection pool".

Thanks!

Relates to:

• https://hibernate.atlassian.net/browse/HSEARCH-2681

I think it's the same problem mentioned here:

• https://discuss.elastic.co/t/elasticsearch-in-docker-and-the-restclient-often-got-java-util-concurrent-timeoutexception/68118

[abeyad]

@javanna can you take a look at this, or otherwise delegate?

[javanna]

thanks @Sanne it is not the first time that I bump into this TimeoutException. Thanks a lot for all the research. Would you like to open a PR with the suggested fix?

[Sanne]

@javanna sure, it's trivial but I'm happy to send a PR. I hope you won't need a reproducer? Developing one to run reliably would take much more time.

[javanna]

@Sanne I see what you mean, send the fix and I will see whether it's possible to write a test for it or not.

[chrkv]

Am I right that this problem is still present in newer versions?

[OuyangKevin]

it is ok in newer versions?

[michaelsogos]

Hi Guys,

Is it still in review? I'm using package version 6.1.0 and i'm getting a lot of java.util.concurrent.TimeoutException.

Can you help me?

[Sanne]

Sorry all I forgot to come back to this, as it's quite hard to make a proper test for it. In my own projects I'd be able to make one but I use very different test helpers, such as Byteman. I'm not sure if it's worth the infrastructure overhead for one test.

The fix is trivial though, if @javanna is ok with it I'll send a PR with the fix only.

For people affected already: workaround is very simple too, it's a one liner of configuration file.
See: https://github.com/hibernate/hibernate-search/pull/1367/files#diff-001ed4d180f5c30deb9754f54284a7d9R150

Essentially set:

.setConnectionRequestTimeout( 0 ) 

This is not to ignore the timeouts on operations but to disable an unusual feature of the Apache HTTP Async Client to timeout on the pool lease after the operation is successfully completed when returning the connection to the pool.
I'm not sure why the Async Client developers have such a feature, but I'm confident it's not the semantics we need in the case of the Elasticsearch client. I figured by debugging the library that zero disables this.

[michaelsogos]

Dear @Sanne ,

It works!!! I didn't encountered that exception anymore.

[rupanjan-baidya]

I have set setConnectionRequestTimeout to 0, but still facing same issue. Elasticsearch version 6.0

	@Override
	protected RestClient createInstance() throws Exception {
		int timeout = 10;
		logger.info("Initiallizing Elasticsearch RestClient");
		RestClientBuilder builder = RestClient.builder(new HttpHost(serverUrl, Integer.parseInt(port.trim()), "http"));
		builder.setMaxRetryTimeoutMillis(timeout * 1000);
		builder.setDefaultHeaders(Constants.HEADERS);
		builder.setFailureListener(new RestClient.FailureListener() {
			@Override
			public void onFailure(HttpHost host) {
				logger.error("FAILURE !!!! FailureListener HAS WOKEN UP!!!!");
			}
		});
		builder.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
			@Override
			public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
				return requestConfigBuilder.setConnectTimeout(timeout * 1000).setSocketTimeout(timeout * 1000)
						.setConnectionRequestTimeout(0);
			}
		});

		RestClient restClient = builder.build();
		if (enableSniffer) {
			this.sniffer = Sniffer.builder(restClient).build();
		}
		logger.info("Initiallized Elasticsearch RestClient");
		return restClient;
	}