Make s3 repository sensitive settings use secure settings

[rjernst]

This change converts repository-s3 to use the new secure settings. In
order to support the multiple ways we allow aws creds to be configured,
it also moves the main methods for the keystore wrapper into a
SecureSettings interface, in order to allow settings prefixing to work.

[rjernst]

I still need to figure out some tests for this, but I wanted to get this up here to get some feedback on how it looks.

[s1monw]

LGTM I don't really know how we should test this. I think the simplest would be to have a class that you can override that adds factory methods for StaticCredentialsProvider etc. instead of using a static method.?

[s1monw]

can't we throw GeneralSecurityException do we have to wrap?

[rjernst]

Sure, we could throw it. I was thinking ahead to Vault integration, but I guess that could make GeneralSecurityExceptions?

[rjernst]

I take it back. Unfortunately, GeneralSecureException is not the only thing that can be thrown. Not all keystore apis throw a subclass of this. There are like 3 different exceptions. :(

[rjernst]

Oops, nevermind. Done!

[rjernst]

@s1monw I found the existing tests for the old settings, made them work with deprecation log checks, and duplicated them to use new secure settings.

[rjernst]

retest this please

[s1monw]

LGTM

[rjernst]

retest this please

[dadoonet]

@rjernst I tried some first tests today:

With elasticsearch 5.1.1

In my config/elasticsearch.yml file:

cloud.aws.access_key: "ACCESS-KEY"
cloud.aws.secret_key: "SECRET-KEY"
repositories.s3.bucket: "test.eu-west-1.elasticsearch.org"
repositories.s3.region: "eu-west-1"

Then I start elasticsearch and run:

curl -XPOST 'http://localhost:9200/_snapshot/backups?pretty=true' -d '{ "type":"s3" }'

This gives:

{
  "acknowledged" : true
}

With elasticsearch 6.0.0

bin/elasticsearch-keystore create
bin/elasticsearch-keystore add cloud.aws.secret_key
# Entering here ACCESS-KEY
bin/elasticsearch-keystore add cloud.aws.access_key
# Entering here SECRET-KEY

In elasticsearch.yml:

repositories.s3.bucket: "test.eu-west-1.elasticsearch.org"
repositories.s3.region: "eu-west-1"

Then I start elasticsearch and run:

curl -XPOST 'http://localhost:9200/_snapshot/backups?pretty=true' -d '{ "type":"s3" }'

This gives:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[backups] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[backups] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object tests-Z8dZ3RULTSKNXfoVU222Rw/master.dat-temp",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: FE35B1FCEC30375A)"
      }
    }
  },
  "status" : 500
}

May be I forgot something though.
I'm going to look at it in debug mode and will update here with findings.

[dadoonet]

Apparently I'm getting back an empty String when running

elasticsearch/plugins/repository-s3/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java

Line 131 in 8015fbb

try (SecureString key = getValue(repositorySettings, settings,

This is then trying to use environment variables, system properties or instance profile credentials. Then it causes that error message.

I don't understand why I'm getting an empty String while asking for repositories.s3.access_key and repositories.s3.secret_key.
Note that I also added those two settings with elasticsearch-keystore app:

$ bin/elasticsearch-keystore list
cloud.aws.access_key
cloud.aws.secret_key
repositories.s3.access_key
repositories.s3.secret_key