elasticsearch-transport will cause stack overflow if host is unreachable and reload_on_failure is true

[cemeyer2]

Code that reproduces:

require 'rspec'
require 'elasticsearch'

describe 'bug in es transport' do
  it 'should not stack overflow' do
    hosts = %w(1.2.3.4 5.6.7.8 9.10.11.12)
    client = Elasticsearch::Client.new hosts: hosts, retry_on_failure: true, reload_connections: true, reload_on_failure: true, transport_options: {request: {timeout: 10}}
    client.cluster.health
  end
end

stack trace:

SystemStackError: stack level too deep
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.1/lib/faraday/connection.rb:406:in `build_exclusive_url'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:191:in `build_env'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/http/faraday.rb:21:in `block in perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/base.rb:187:in `call'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/base.rb:187:in `perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/sniffer.rb:31:in `block in hosts'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/sniffer.rb:30:in `hosts'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/base.rb:71:in `reload_connections!'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/base.rb:198:in `rescue in perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/base.rb:177:in `perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/sniffer.rb:31:in `block in hosts'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/sniffer.rb:30:in `hosts'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/base.rb:71:in `reload_connections!'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/base.rb:198:in `rescue in perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/base.rb:177:in `perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
/Users/cmeyer/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.9/lib/elasticsearch/transport/transport/sniffer.rb:31:in `block in hosts'
...
...

[karmi]

Hi, can you verify it's still the case?

I've changed the code slightly -- hosts = %w(foo bar baz) -- and get this:

karmi▹ time rspec ~/Desktop/rspec_issue.rb 
F

Failures:

  1) bug in es transport should not stack overflow
     Failure/Error: client.cluster.health
     Faraday::ConnectionFailed:
       getaddrinfo: nodename nor servname provided, or not known
     # /Users/karmi/Desktop/rspec_issue.rb:8:in `block (2 levels) in <top (required)>'

Finished in 7.28 seconds (files took 0.34529 seconds to load)
1 example, 1 failure

Failed examples:

rspec /Users/karmi/Desktop/rspec_issue.rb:5 # bug in es transport should not stack overflow


real    0m7.875s
user    0m5.402s
sys 0m0.811s

(I'm really sorry for such a long, ridiculous delay with the reply.)

[shaicoleman]

I can reproduce 1.0.14, Ruby 2.2.2, Faraday 0.9.2 on Ubuntu 14.04, with:
retry_on_failure = true, reload_connections = true, reload_on_failure = default, randomize_hosts = true

Backtrace:

/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/ipaddr.rb:528 :in `each`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/ipaddr.rb:528 :in `inject`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/ipaddr.rb:528 :in `in_addr`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/ipaddr.rb:490 :in `initialize`
[GEM_ROOT]/gems/domain_name-0.5.25/lib/domain_name.rb:87 :in `new`
[GEM_ROOT]/gems/domain_name-0.5.25/lib/domain_name.rb:87 :in `initialize`
[GEM_ROOT]/gems/http-cookie-1.0.2/lib/http/cookie_jar/hash_store.rb:70 :in `new`
[GEM_ROOT]/gems/http-cookie-1.0.2/lib/http/cookie_jar/hash_store.rb:70 :in `each`
[GEM_ROOT]/gems/http-cookie-1.0.2/lib/http/cookie_jar.rb:163 :in `each`
[GEM_ROOT]/gems/http-cookie-1.0.2/lib/http/cookie_jar.rb:131 :in `each`
[GEM_ROOT]/gems/http-cookie-1.0.2/lib/http/cookie_jar.rb:131 :in `sort`
[GEM_ROOT]/gems/http-cookie-1.0.2/lib/http/cookie_jar.rb:131 :in `cookies`
[GEM_ROOT]/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:31 :in `cookies`
[GEM_ROOT]/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:49 :in `cookie_value`
[GEM_ROOT]/gems/httpclient-2.6.0.1/lib/httpclient.rb:1131 :in `create_request`
[GEM_ROOT]/gems/httpclient-2.6.0.1/lib/httpclient.rb:963 :in `do_request`
[GEM_ROOT]/gems/httpclient-2.6.0.1/lib/httpclient.rb:822 :in `request`
[GEM_ROOT]/gems/faraday-0.9.2/lib/faraday/adapter/httpclient.rb:36 :in `call`
[GEM_ROOT]/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139 :in `build_response`
[GEM_ROOT]/gems/faraday-0.9.2/lib/faraday/connection.rb:377 :in `run_request`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:21 :in `block in perform_request`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191 :in `call`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191 :in `perform_request`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20 :in `perform_request`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:32 :in `block in hosts`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:89 :in `block in timeout`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:99 :in `call`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:99 :in `timeout`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:31 :in `hosts`

<repeat>
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:75 :in `reload_connections!`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:66 :in `get_connection`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:183 :in `perform_request`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20 :in `perform_request`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:32 :in `block in hosts`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:89 :in `block in timeout`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:99 :in `call`
/usr/local/rvm/rubies/ruby-2.2.2/lib/ruby/2.2.0/timeout.rb:99 :in `timeout`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:31 :in `hosts`
</repeat>

[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:75 :in `reload_connections!`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:66 :in `get_connection`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:183 :in `perform_request`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20 :in `perform_request`
[GEM_ROOT]/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/client.rb:119 :in `perform_request`
[GEM_ROOT]/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/search.rb:167 :in `search`

Can also reproduce with the rspec above with your hosts modification

Faraday::TimeoutError:
  execution expired
# /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:80:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:21:in `block in perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191:in `call'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:32:in `block in hosts'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'

<repeat>
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:75:in `reload_connections!'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:202:in `rescue in perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:181:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:32:in `block in hosts'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'
</repeat>

# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:75:in `reload_connections!'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:202:in `rescue in perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:181:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/client.rb:119:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/namespace/common.rb:21:in `perform_request'
# /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/cluster/health.rb:49:in `health'

[karmi]

When I run (notice I've fixed the timeout options and decreased the timeout to 1 second):

require 'rspec'
require 'elasticsearch'

describe 'bug in es transport' do
  it 'should not stack overflow' do
    hosts = %w(1.2.3.4 5.6.7.8 9.10.11.12)
    client = Elasticsearch::Client.new hosts: hosts, retry_on_failure: true, reload_connections: true, reload_on_failure: true, transport_options: { request: { open_timeout: 1, timeout: 1 } }
    client.cluster.health
  end
end

I get:

karmi▹ time rspec ~/Desktop/rspec_issue.rb 
F

Failures:

  1) bug in es transport should not stack overflow
     Failure/Error: client.cluster.health
     Net::OpenTimeout:
       execution expired
     # ./lib/elasticsearch/api/namespace/common.rb:21:in `perform_request'
     # ./lib/elasticsearch/api/actions/cluster/health.rb:49:in `health'
     # /Users/karmi/Desktop/rspec_issue.rb:8:in `block (2 levels) in <top (required)>'

Finished in 6.07 seconds (files took 0.31101 seconds to load)
1 example, 1 failure

Failed examples:

rspec /Users/karmi/Desktop/rspec_issue.rb:5 # bug in es transport should not stack overflow


real    0m6.545s
user    0m0.437s
sys 0m0.085s

... which is what should happen, and the "stack level too deep" doesn't happen. So you can reproduce the "stack level too deep" error, @shaicoleman ?

[shaicoleman]

The root cause of the issue is still there, and running the above gives me 800 lines of a recrusive stack trace as above, and the test takes a total of 75 seconds for a single request.

  1) bug in es transport should not stack overflow
     Failure/Error: client.cluster.health
     Net::OpenTimeout:
       execution expired
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/connection.rb:406:in `build_exclusive_url'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:192:in `build_env'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:21:in `block in perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191:in `call'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:191:in `perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:32:in `block in hosts'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:75:in `reload_connections!'

[800 lines snipped]

     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:75:in `reload_connections!'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:202:in `rescue in perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:181:in `perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:32:in `block in hosts'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:75:in `reload_connections!'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:202:in `rescue in perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/base.rb:181:in `perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-transport-1.0.14/lib/elasticsearch/transport/client.rb:119:in `perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/namespace/common.rb:21:in `perform_request'
     # /home/shai/.rvm/gems/ruby-2.2.2/gems/elasticsearch-api-1.0.14/lib/elasticsearch/api/actions/cluster/health.rb:49:in `health'
     # ./elasticsearch_spec.rb:7:in `block (2 levels) in <top (required)>'

Finished in 1 minute 14.33 seconds (files took 0.15137 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./elasticsearch_spec.rb:4 # bug in es transport should not stack overflow```

[karmi]

I still cannot reproduce it :(

$ruby -v
ruby 2.2.2p95 (2015-04-13 revision 50295) [x86_64-darwin14]

$ time rspec ~/Desktop/rspec_issue.rb
F

Failures:

  1) bug in es transport should not stack overflow
     Failure/Error: client.cluster.health
     Net::OpenTimeout:
       execution expired
     # /Users/karmi/Desktop/rspec_issue.rb:8:in `block (2 levels) in <top (required)>'

Finished in 18.22 seconds (files took 0.82732 seconds to load)
1 example, 1 failure

Failed examples:

rspec /Users/karmi/Desktop/rspec_issue.rb:5 # bug in es transport should not stack overflow

real    0m19.366s

[shaicoleman]

I've emailed you login details to an instance where you'll be able to reproduce the issue

[karmi]

Returning to this old issue, I've tried on the instance Shai provided, and got an "cannot allocate memory" error, I'll try this out on a Linux machine, because it's not reproducible on MaC OS X...

[karmi]

Returning to the old thread again, almost a year later :), I've tried on Ubuntu 16.04.1 LTS (xenial), and couldn't reproduce the issue:

sudo apt install ruby-dev
sudo gem install rspec
sudo gem install elasticsearch

# rspec_issue.rb
require 'rspec'
require 'elasticsearch'

describe 'bug in es transport' do
  it 'should not stack overflow' do
    hosts = %w(1.2.3.4 5.6.7.8 9.10.11.12)
    client = Elasticsearch::Client.new hosts: hosts, retry_on_failure: true, reload_connections: true, reload_on_failure: true, transport_options: { request: { open_timeout: 1, timeout: 1 } }
    client.cluster.health
  end
end

$ rspec rspec_issue.rb
F

Failures:

  1) bug in es transport should not stack overflow
     Failure/Error: client.cluster.health

     Net::OpenTimeout:
       execution expired
     # /var/lib/gems/2.3.0/gems/faraday-0.11.0/lib/faraday/connection.rb:406:in `build_exclusive_url'
     # ...
     # /var/lib/gems/2.3.0/gems/elasticsearch-api-5.0.0/lib/elasticsearch/api/actions/cluster/health.rb:58:in `health'
     # ./rspec_issue.rb:8:in `block (2 levels) in <top (required)>'
     # ------------------
     # --- Caused by: ---
     # Net::OpenTimeout:
     #   execution expired
     #   /var/lib/gems/2.3.0/gems/faraday-0.11.0/lib/faraday/adapter/net_http.rb:78:in `perform_request'

Finished in 2.04 seconds (files took 0.16988 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./rspec_issue.rb:5 # bug in es transport should not stack overflow

Can you guys check with the a new gem version, or the master branch? I don't wanna ignore a potential bug here, but I need to reproduce...

[shaicoleman]

Still happening for me with the master branch.

[shaicoleman]

I've created a Dockerfile to replicate the issue
https://github.com/shaicoleman/es-bug

[karmi]

I've debugged the issue on a Ubuntu 16.04 LTS in AWS, and have found a way to consistently make the test succeed or fail.

It consistently fails when the client is initialized with the default HTTP adapter, ie. Ruby's built-in Net::HTTP. For all the other libraries I have used: Typhoeus, Patron and Curb, the script correctly exits after the configured timeout. So there is something wrong when using Net::HTTP, but I haven't found the cause.

Given that people should use a high-performance library in production (such as those mentioned), I think the issue has not much gravity.

In terms of the Ruby code, this is all should be needed for the script to exit correctly, and not with an overflow:

require 'typhoeus'
require 'typhoeus/adapters/faraday'

In terms of the environment, I have changed the Docker image configuration (and the script) in the following way, to emulate what I am seeing on the bare server:

diff --git a/Dockerfile b/Dockerfile
index 66669e2..5435dfb 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -5,7 +5,8 @@ ENV DEBIAN_FRONTEND=noninteractive
 
 RUN \
   apt-get update && \
-  apt-get install -y eatmydata wget curl software-properties-common sudo htop
+  apt-get install -y eatmydata wget curl software-properties-common sudo htop && \
+  apt-get install -y build-essential libssl-dev libreadline-dev zlib1g-dev libcurl4-openssl-dev
 
 ENV LD_PRELOAD /usr/lib/x86_64-linux-gnu/libeatmydata.so
 
diff --git a/Gemfile b/Gemfile
index fae10ca..db48619 100644
--- a/Gemfile
+++ b/Gemfile
@@ -1,3 +1,4 @@
 source 'https://rubygems.org'
 
+gem 'typhoeus'
 gem 'elasticsearch'
diff --git a/es-bug.rb b/es-bug.rb
index c05d965..e158f1b 100644
--- a/es-bug.rb
+++ b/es-bug.rb
@@ -1,5 +1,9 @@
+require 'typhoeus'
+require 'typhoeus/adapters/faraday'
+
 require 'elasticsearch'
 
 hosts = %w(1.2.3.4 5.6.7.8 9.10.11.12)
 client = Elasticsearch::Client.new hosts: hosts, retry_on_failure: true, reload_connections: true, reload_on_failure: true, transport_options: { request: { open_timeout: 1, timeout: 1 } }
-client.cluster.health
+puts client.transport.connections.first.connection.builder.handlers.inspect
+puts client.cluster.health

Please try it out and let me know.