TransportClient behavior when the server node is not available

[tkurki]

We have experienced ES client jamming the jvm and the whole virtual server when the ES server is intermittently not reachable (either down on purpose or with a flaky connection).

When the issue bites we end up with thousands of threads, which will cause OutOfMemory, but the huge number of threads in the generic threadpool exhausts OS resources and the whole virtual server becomes unusable. If there is no previous terminal session that allows killing the offending JVM process we must reboot the whole server.

ES version 0.90.7.

[tkurki]

Thread dump of the problem in action: https://gist.github.com/tkurki/9066012

[tkurki]

Same type of problem indicated here:
https://groups.google.com/forum/#!msg/elasticsearch/oKQMoOKhSU8/09-P9bU4t_wJ
#1930

[tkurki]

It may be that the issue is already fixed:
c324251#diff-359977d2ee02995c3dea93913c31d4b3L255

We will try 0.90.11 next. Upgrade to 1.0 series is not possible immediately because of inhouse rivers.

[timorantalaiho]

I am intrigued by the fencepost change that @tkurki linked, because the comment of the commit and the issue it closes seem to be about a monitoring enhancement, and the change to the if condition seems to be completely unrelated to that issue, but it might have fixed the race condition problem we are experiencing. @bleskes , do you remember anything about what triggered you to do that change?

[bleskes]

I vaguely recall running into a racing condition during the development of that commit. Though I don't recall the details it had to do with the a connect exception both being picked up by the RetryListener as a callback here: c324251#diff-359977d2ee02995c3dea93913c31d4b3L259 and explictly called in a catch clause afterwards. A >= check is anyway safer then == in this case.

@tkurki can report on how it goes with 0.90.11?

[tkurki]

After upgrading to 0.90.11 we have not experienced this issue - the same test setup works now reliably. Apparently the fix for #4162 fixed this.

[javanna]

Related to the >= check mentioned above, it is indeed safer thatn ==. The fact that the retry counts can even get higher than the actual number of connected nodes is that the retry listener gets notified multiple times for the same failure. See #6829 .