Unbound threadpools considered harmful

[tkurki]

ElasticSearch server and java client both have an unbound, undocumented threadpool that can cause lockup of the whole JVM process and take down the whole server machine: #5151.

The minimum change is documenting the unbound generic pool & instructions for reconfiguring it at http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-threadpool.html.

IMHO unbound threadpool is a Bad Idea from the start. Bound pool with some reasonable size queue and abort policy would not cause jvm/server lockup in a runaway case, instead you would get some meaningful information in the log.

[tkurki]

https://groups.google.com/forum/#!msg/elasticsearch/oKQMoOKhSU8/09-P9bU4t_wJ

[scottbessler]

This appears to have bit us as well. We noticed it when we had an ActionListener on a ListenableActionFuture<BulkResponse> who took perhaps too long to execute and/or had concurrency issues (it was actually firing a Guava Eventbus event where the handlers are synchronized). We started seeing exceptions like:

org.elasticsearch.client.transport.NoNodeAvailableException: No node available
    at org.elasticsearch.client.transport.TransportClientNodesService$RetryListener.onFailure(TransportClientNodesService.java:249)

And then with Yourkit attached we saw the same symptom of 7000+ of these threads being spun up, and hanging the jvm.

[clintongormley]

@scottbessler What version of Elasticsearch are you using?

[kimchy]

The idea of the generic thread pool is to have an option to fork to a thread without having to block on it or get rejected, which is needed in some cases. In places where it is used, its good, but there are places where it simply should not be used, specifically, the listening thread pool when the pool is threaded, we should have a dedicated thread pool for it that is bounded.

[scottbessler]

0.90.6

[clintongormley]

@scottbessler It appears to be fixed in 0.90.8 onwards #5151 #4162

[scottbessler]

The race condition may be fixed, but the unbounded undocumented thread pool lurks waiting for a similar issue to crush JVMs. If nothing else it seems that at least letting users configure this threadpool (even if the default is unbounded) would at least let them choose their poison.