Support file compression before uploading to s3

[adamvduke]

It would be valuable to have the ability to configure the plugin to compress files before uploading them to s3. At the moment we manually download, compress, upload, delete uncompressed files. Perhaps something like:

s3  {
  ...
  compress => 'gzip'
}

This is a feature I'm considering implementing, but I wanted to get a sense of if the maintainers would find this valuable, and/or if there are any other concerns I should be aware of before taking it on.

I see that #1 is a fairly extensive refactor. Is there a timeline for when that is expected to be merged? I'd like to avoid having to rebase against a large change.

[ph]

@adamvduke I'll try to push #1 to be merged when everyone get back from the holidays.
And the changes are major but was needed and it fix a few issues as well.

[ph]

I agree to be able to push a gzip file directly to S3 would be a nice feature to have. I think this should be a pluggable interface. I had prior discussion with the team if this feature should be implemented as a codec or not, I'll try to find the discussion and add it here.

For the timeline, it should be released shortly after 1.5 GA and with the new version you will be able to update the plugins individually.

[adamvduke]

@ph Sounds great. Have a good holiday!

[oroce]

Any update on this? Is it decided whether gzip should be a codec or not?
I have a working solution for 1.4: https://github.com/oroce/logstash-output-s3/blob/gzip-configurable-dir/lib/logstash/outputs/s3.rb#L378-L385
But I would love to upgrade to 1.5 and even add gzip support to this module.

[f1ren]

@oroce I was not able to use your code.
In debug mode I got
S3: file: ls.s3.erans-mini-2.2015-04-16T11.51.tag_s3.part0.txt was put on bucket test-logstash {:level=>:debug, :file=>"logstash/outputs/s3.rb", :line=>"250"}
Without any file in my S3 bucket.
The local temporary file remained empty as well.

[oroce]

@navatm hm i was using that module for like 2 months with the following configuration without any problems:

output {
    stdout { codec => rubydebug }
    s3 {
        bucket => '<bucket>'
        access_key_id => '<access>'
        endpoint_region => 'eu-west-1'
        format => 'json'
        secret_access_key => '<secret>'
        #time_file => 5
        size_file => 504800
        gzip => true
    }
}

Sadly I'm not using it anymore, because I implemented it just for one of my sideprojects.

[shribigb]

@ph When this is going to be merged? I would like to have this compression support.

[f1ren]

@shribigb we've been working with https://github.com/adience-code/logstash-output-s3 on production for a month. No errors yet. We write ~100GB a day, compress and flush every 4 hours.

[shaharmor]

+1

[f1ren]

Since your name is familiar, I just added a pull request.

On Wed, Jul 29, 2015, 17:33 Shahar Mor notifications@github.com wrote:

+1

—
Reply to this email directly or view it on GitHub
#3 (comment)
.

[vpack]

+1

[reid-me]

+1

[robinup]

+1