upload stops working until restarted #79
Description
I was running Logstash v1.5.6 with output plugin (with arguably minor modifications vs core-v1 https://github.com/ihorkhavkin/logstash-output-s3/tree/v1-patched ) and hitting error like this:
{:timestamp=>"2016-05-10T17:58:43.262000+0000", :message=>"upload_worker exception", :ex=>["org/jruby/ext/openssl/SSLSocket.java:768:in `syswrite'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jruby-openssl-0.9.12-java/lib/jopenssl19/openssl/buffering.rb:318:in `do_write'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jruby-openssl-0.9.12-java/lib/jopenssl19/openssl/buffering.rb:336:in `write'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:199:in `write0'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:173:in `write'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:190:in `writing'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/protocol.rb:172:in `write'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:1964:in `send_request_with_body_stream'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:1930:in `exec'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/http/patch.rb:78:in `new_transport_request'", "org/jruby/RubyKernel.java:1242:in `catch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/http/patch.rb:77:in `new_transport_request'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:1302:in `request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/http/connection_pool.rb:356:in `request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/http/net_http_handler.rb:64:in `handle'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/http/connection_pool.rb:131:in `session_for'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/http/net_http_handler.rb:56:in `handle'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:253:in `make_sync_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:289:in `retry_server_errors'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/s3/region_detection.rb:11:in `retry_server_errors'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:249:in `make_sync_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:509:in `client_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:391:in `log_client_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:477:in `client_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:373:in `return_or_raise'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/core/client.rb:476:in `client_request'", "(eval):3:in `put_object'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:1765:in `write_with_put_object'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-v1-1.66.0/lib/aws/s3/s3_object.rb:611:in `write'", "/opt/logstash/vendor/local_gems/dd42f719/logstash-output-s3-1.0.2.3-20160126-1615-643afc43/lib/logstash/outputs/s3.rb:155:in `write_on_bucket'", "org/jruby/RubyIO.java:1183:in `open'", "/opt/logstash/vendor/local_gems/dd42f719/logstash-output-s3-1.0.2.3-20160126-1615-643afc43/lib/logstash/outputs/s3.rb:151:in `write_on_bucket'", "/opt/logstash/vendor/local_gems/dd42f719/logstash-output-s3-1.0.2.3-20160126-1615-643afc43/lib/logstash/outputs/s3.rb:255:in `move_file_to_bucket'", "/opt/logstash/vendor/local_gems/dd42f719/logstash-output-s3-1.0.2.3-20160126-1615-643afc43/lib/logstash/outputs/s3.rb:406:in `upload_worker'", "/opt/logstash/vendor/local_gems/dd42f719/logstash-output-s3-1.0.2.3-20160126-1615-643afc43/lib/logstash/outputs/s3.rb:384:in `configure_upload_workers'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.21/lib/stud/task.rb:12:in `initialize'"], :level=>:error}
It looks like something that might affect even the current master as my changes shouldn't affect exception handling (except for logging this specific case that has blown). I suspect that when upload to S3 fails (e.g. somewhere in the middle of upload due to intermittent networking or s3 issue), uploader just exits and eventually no uploader is running at all.
In advance, sorry for nasty backtrace from modified logstasg-output-s3 without exception name . But any thoughts are welcome. At this point I don't really understand how logstash-output-s3 current master is supposed to handle dying workers in case of S3 failure. If this is common issue, it might be worth looking into.