AWS Credentials " A key must be specified"

[Jinkxed]

Please post all product and debugging questions on our forum. Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

For all general issues, please provide the following details for fast resolution:

• Version: 4.0.3
• Operating System: Amazon Linux
• Config File (if you have sensitive info, please remove it):

        s3 {
            region    => "us-east-1"
            prefix    => "someteam/%{+YYYY.MM.dd}-"
            bucket    => "some-events"
          }

• Sample Data:
• Steps to Reproduce:

We are using IAM roles, but I have environment variables with the values stored as well. I've tried setting the keys manually. I've tried using environment variables. I've tried removing them all together.

I've also tried with a basic prefix, and nothing seems to work. Here is the error I keep getting:

[2017-01-16T23:56:12,085][ERROR][logstash.outputs.s3      ] Uploading failed, retrying {:exception=>#<Aws::S3::Errors::InvalidRequest: A key must be specified>, :path=>"/tmp/logstash/ls.s3.prod-graylog-processor-1880aa0f.2017-01-13T15.38.part0.txt", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_accelerate.rb:33:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/request.rb:70:in `send_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/base.rb:207:in `create_multipart_upload'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/multipart_file_uploader.rb:52:in `initiate_upload'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/multipart_file_uploader.rb:43:in `upload'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:32:in `upload'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/object.rb:251:in `upload_file'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.3/lib/logstash/outputs/s3/uploader.rb:38:in `upload'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.3/lib/logstash/outputs/s3/uploader.rb:29:in `upload_async'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/concurrent-ruby-1.0.0-java/lib/concurrent/executor/java_executor_service.rb:94:in `run'", "Concurrent$$JavaExecutorService$$Job_754842726.gen:13:in `run'"]}

[ph]

I've tested the master current master of the s3 plugin with the following configuration:

input {
    generator {
    }
}

output {
    s3 {
        region    => "us-east-1"
        prefix    => "someteam/%{+YYYY.MM.dd}-"
        bucket    => "ls-ph-test"
        }
}

And the following environment variables:

AWS_ACCESS_KEY_ID=<redacted>
AWS_SECRET_ACCESS_KEY=<redacted>

I was able to send event to s3.

[Jinkxed]

I have them both set oddly enough. I also tried setting them in the output. Neither worked. I'll try your test and see if I get a different result.

[ph]

OK, I think I know what is going on, If we look at the path from the log statement.

 :path=>"/tmp/logstash/ls.s3.prod-graylog-processor-1880aa0f.2017-01-13T15.38.part0.txt",

This kind of local filepath structure is created by the s3 output pre 4.0.0 and the newer code doesn't know how to handle that old format. It doesn't know because it doesn't expect to be present.

I suspect the following happened, you ran logstash with an older plugin version and it crash or it stop with remaining events in temporary file on disk that were never uploaded to s3.

By default the 4.0.0 plugin will try to upload any remaining file from the temporary directory, it's the restore_from_crash option. But with the older format it doesn't know how to generate the S3 Key on the bucket and the plugin fails.

in conclusion:

• Your credentials appear to work, since the error mean we are trying to upload without a key.
• Upload manually the files from the temporary folder /tmp/logstash to S3, remove any files in that directory and restart Logstash that should clean your situation.

[ph]

@sc-chad I haven't tested IAMRoles but I believe they should work with the error that you have added in this issue description.

[Jinkxed]

@ph trying the manual upload / clear old files route. That sounds like what is going on and I very much appreciate you looking into it. A follow up question, does it matter if I have more than 1 logstash processor uploading to the same s3 bucket?

[ph]

@sc-chad there is an issue with the current version of the output see #123, I will get that released soon :)

@sc-chad Nothing prevents to have more than one indexer sending to the same bucket; there is currently no check to see if the key is already present, But the clash risk should be minimal; each file has a UUID + a Date + a part number.

BUT, Nothing prevents you to add the indexer to the dynamic prefix that you are using on s3, something like:

prefix => "someteam/%{+YYYY.MM.dd}/%{indexer_name]"

Note that I am assuming that the event has a indexer_name field.

[Jinkxed]

@ph Okay awesome! I uploaded the old files manually to another folder, cleared /tmp/logstash and restarted logstash and everything is now running again!

I'll keep an eye out on open files for the other issue and see if its still happening.

thank you very much for your help!