add support for force_path_style option

[jsvd]

this is useful when using custom endpoints in s3

replaces #139

a similar PR should be made against https://github.com/logstash-plugins/logstash-output-s3

[jakelandis]

is this an AWS::S3 flag ? I can't find it documented .... do you have a link ?

[jsvd]

The option can be found here: https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html

Btw, the constructor validates that the settings exist, so force_path_style has to be valid (we'd get an exception otherwise).

[jakelandis]

@jsvd - thanks, sometimes i think i am blind.

There are alot of good settings in there... should we open it up the configuration to general purpose hash of options to pass in (kinda like we do with jdbc and sequel ) ?

[jsvd]

It's a good question, we could change that but would be a breaking change, maybe we can open an issue to do it until logstash 7.0 GAs

[jakelandis]

is it a breaking change ?

config :additional_settings, :validate => :hash, :default => {}
...
options.merge!(@additional_settings)

[jsvd]

I'm thinking that probably we'd like to remove some of the current config settings as they'd be done through the additional_settings hash.

[jakelandis]

we could just swap the merge to be result = @additional_settings.merge(options) calling it additional implies that other things in there take precedence.

[jsvd]

ok. I have no strong feelings here, will redo as generic additional_settings

[jakelandis]

LGTM
EDIT: probably want to clean up the commit logs.

[jsvd]

thanks! @jakelandis

[SalmaanP]

Just a quick question, how would you connect to an custom endpoint. Which variable should the url go in? Also can SSL options like certificates be passed in the additional_settings?
Example, is this correct?:

    input {
      s3 {
        "access_key_id" => "1234",
        "secret_access_key" => "secret",
        "bucket" => "bucket-name",
        "additional_settings" => {
          "force_path_style => true,
          "endpoint" => "https://domain" (?)
        }
      }

[webmat]

@SalmaanP Custom endpoints are supported at the top level since v3.3.0, see doc:

      s3 {
        "access_key_id" => "1234",
        "secret_access_key" => "secret",
        "bucket" => "bucket-name",
        "endpoint" => "https://s3.dualstack.us-east-1.amazonaws.com"
        #...

Remember that you must include the protocol, not just the domain name. See the full list of S3 endpoints.

Note also that this does not guarantee you'll be able to connect to "compatible" endpoints from S3 competitors. This plugin uses the Ruby AWS SDK, which doesn't guarantee this use case. This setting can be useful for IPV6 (example above) or for govcloud, however.

[SalmaanP]

@webmat I see endpoint is there in the documentation but I don't see it defined anywhere here

In my tests with a custom on-prem s3 bucket, using the latest version of the plugin and endpoint at the top level did not work, so did putting it in additional options.

However, using it by changing the plugin as done here did work. I was also able to add SSL options here in the same way and it started to work.

[jsvd]

This functionality is provided by the logstash-mixin-aws, in version 4.3.0 https://github.com/logstash-plugins/logstash-mixin-aws/blob/master/CHANGELOG.md#430

Make sure you have upgraded both s3 and this mixing. To see plugins and versions installed you can do bin/logstash-plugin list --verbose

[SalmaanP]

@jsvd I see, thanks for the information!