Simplify aws configuration

[rjernst]

While working on #22479, I noticed that we have quite a few different ways to specify connection configuration for s3 repositories. There are 4 different setting names for each "real" setting (eg access key) that I counted, which are tried successively as fallbacks in a web of Setting objects. Can we simplify the configuration to just one set of settings?

Perhaps we can have named configurations, and the name "default" would be special? For example:

aws.configuration.default:
    access_key: foo
    secret_key: bar
    proxy_host: example.com
    proxy_port: 1234

Now if you replace default with myconfig, then in an s3 repository, we can use say configuration: myconfig. This would greatly simplify settings parsing, as there would be one place to look, and it still allows users without complicated setups to not need fancy configuration (just setup the default configuration, no need to specify anything within a repository, similar today to setting eg cloud.aws.access_key).

[s1monw]

++

[s1monw]

#22526 and #22523 might be helpful here.

[dadoonet]

I think that's a great idea. I did something similar for azure but I'm using another way to define things:

If you have only one account, you define:

cloud.azure.storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1

Then when you create the repository, you don't need to reference which account you want to use as there is only one.

# my_account1 will be used
PUT _snapshot/my_backup
{
    "type": "azure"
}

If you need multiple accounts, you explicitly set which is the default one with default: true:

cloud.azure.storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2

Again, if you create a repository without setting the account you want to use the one which is marked as default will be used.

# my_account1 will be used
PUT _snapshot/my_backup
{
    "type": "azure"
}

Otherwise, you have to explicitly set the account to use:

# my_account2 will be used
PUT _snapshot/my_backup
{
    "type": "azure",
    "settings": {
        "account": "my_account2"
    }
}

I'm fine doing it as you are proposing but in that case I'd open a follow up issue to apply the same pattern to azure plugin so we are consistent across repository plugins.

Also if we go your way, as for naming, may be we can use something like _default_ instead:

aws.configuration._default_:
    access_key: foo
    secret_key: bar
    proxy_host: example.com
    proxy_port: 1234

We are using some specific names like this for the network layer for instance.

WDYT?

[rjernst]

If you need multiple accounts, you explicitly set which is the default one with default: true

I don't like having a flag, as this requires validation code to ensure two accounts were not set with default: true. I think a special name is just fine.

Also if we go your way, as for naming, may be we can use something like default instead

I also don't like having to use special characters. I don't see any reason the special configuration name default can't be used.

[dadoonet]

I don't like having a flag, as this requires validation code to ensure two accounts were not set with default: true. I think a special name is just fine.

I'm ok with your way of doing it. Just saying that I want to be consistent across plugins so I will move azure to use the same kind of "global" configuration which can be referenced then later.

I don't see any reason the special configuration name default can't be used.

Well. It's just a way to make sure and clearly identify what are reserved words.
Like what we have in the rest of elasticsearch. _all field is not all field.
GET index/_search instead of GET index/search.

Even more: https://www.elastic.co/guide/en/elasticsearch/reference/5.1/default-mapping.html

PUT my_index
{
  "mappings": {
    "_default_": { 
      "_all": {
        "enabled": false
      }
  }
}

So _default_ seems a better name IMO but I let other people comment. Not a big deal to me.

[rjernst]

It's just a way to make sure and clearly identify what are reserved words.

It's not really a reserved word though. It is just the the default configuration name when reading the config setting in repository settings.

[dadoonet]

Then, let's do something like:

aws.configuration:
    foo:
        access_key: foo
        secret_key: bar
        proxy_host: example.com
        proxy_port: 1234
aws.default: foo

Or something along those lines?
We can define that aws.default default value is default which allows your way of writing the settings:

aws.configuration:
    default:
        access_key: foo
        secret_key: bar
        proxy_host: example.com
        proxy_port: 1234

[rjernst]

We can define that aws.default default value is default

That is just an extra level of obfuscation with an additional setting that adds no benefit.