Socket permission on S3 Repository creation

[dadoonet]

Elasticsearch version: 6.0.0-alpha1-SNAPSHOT (872412f)
Plugins installed: [ repository-s3 ]
JVM version: 1.8.0_121
OS version: MacOS
Description of the problem including expected versus actual behavior:

Registering an S3 repository is now failing on master branch.

Steps to reproduce:

1. Install the distribution, and the repository-s3 plugin
2. Create the keystore and s3 credentials:

$ bin/elasticsearch-keystore list
s3.client.my_config.access_key
s3.client.my_config.secret_key

3. Starts elasticsearch
4. Register a repository:

curl -X POST 'http://127.0.0.1:9200/_snapshot/backups?pretty=true&error_trace=true' -H 'Content-Type: application/json' -d '{
   "type":"s3",
   "settings":{
      "bucket": "BUCKET",
      "region": "eu-west-1"
   }
}'

Provide logs (if relevant):

[2017-02-20T19:54:01,577][INFO ][o.e.n.Node               ] [] initializing ...
[2017-02-20T19:54:01,640][INFO ][o.e.e.NodeEnvironment    ] [Qo1UYJ1] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [11.7gb], net total_space [464.7gb], spins? [unknown], types [hfs]
[2017-02-20T19:54:01,641][INFO ][o.e.e.NodeEnvironment    ] [Qo1UYJ1] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-02-20T19:54:01,642][INFO ][o.e.n.Node               ] node name [Qo1UYJ1] derived from node ID [Qo1UYJ1SSKy_sEz70V1S3w]; set [node.name] to override
[2017-02-20T19:54:01,645][INFO ][o.e.n.Node               ] version[6.0.0-alpha1-SNAPSHOT], pid[69444], build[872412f/2017-02-20T18:38:31.989Z], OS[Mac OS X/10.12.3/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-02-20T19:54:01,646][WARN ][o.e.n.Node               ] version [6.0.0-alpha1-SNAPSHOT] is a pre-release version of Elasticsearch and is not suitable for production
[2017-02-20T19:54:03,618][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [aggs-matrix-stats]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [ingest-common]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [lang-expression]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [lang-mustache]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [lang-painless]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [percolator]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [reindex]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [repository-url]
[2017-02-20T19:54:03,619][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded module [transport-netty4]
[2017-02-20T19:54:03,620][INFO ][o.e.p.PluginsService     ] [Qo1UYJ1] loaded plugin [repository-s3]
[2017-02-20T19:54:06,880][INFO ][o.e.n.Node               ] initialized
[2017-02-20T19:54:06,880][INFO ][o.e.n.Node               ] [Qo1UYJ1] starting ...
[2017-02-20T19:54:07,495][INFO ][o.e.t.TransportService   ] [Qo1UYJ1] publish_address {127.0.0.1:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {127.0.0.1:9300}
[2017-02-20T19:54:10,593][INFO ][o.e.c.s.ClusterService   ] [Qo1UYJ1] new_master {Qo1UYJ1}{Qo1UYJ1SSKy_sEz70V1S3w}{ByBFhbOGTr2rsilGvWAygQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-02-20T19:54:10,637][INFO ][o.e.h.n.Netty4HttpServerTransport] [Qo1UYJ1] publish_address {127.0.0.1:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {127.0.0.1:9200}
[2017-02-20T19:54:10,652][INFO ][o.e.n.Node               ] [Qo1UYJ1] started
[2017-02-20T19:54:10,654][INFO ][o.e.g.GatewayService     ] [Qo1UYJ1] recovered [0] indices into cluster_state
[2017-02-20T19:54:24,971][WARN ][o.e.r.RepositoriesService] [Qo1UYJ1] failed to create repository [s3][backups]
java.security.AccessControlException: access denied ("java.net.SocketPermission" "169.254.169.254:80" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
        at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:1051) ~[?:1.8.0_121]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:510) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) ~[?:?]
        at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:91) ~[?:?]
        at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) ~[?:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ~[?:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) ~[?:?]
        at org.elasticsearch.cloud.aws.InternalAwsS3Service.client(InternalAwsS3Service.java:70) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:345) ~[?:?]
        at org.elasticsearch.plugin.repository.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:71) ~[?:?]
        at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:383) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:356) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:679) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:658) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:617) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:1117) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
[2017-02-20T19:54:24,981][WARN ][o.e.r.RepositoriesService] [Qo1UYJ1] failed to create repository [backups]
org.elasticsearch.repositories.RepositoryException: [backups] failed to create repository
        at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:388) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:356) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:679) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:658) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:617) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:1117) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:527) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: java.security.AccessControlException: access denied ("java.net.SocketPermission" "169.254.169.254:80" "connect,resolve")
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_121]
        at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_121]
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_121]
        at java.lang.SecurityManager.checkConnect(SecurityManager.java:1051) ~[?:1.8.0_121]
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:510) ~[?:?]
        at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:308) ~[?:?]
        at sun.net.www.http.HttpClient.New(HttpClient.java:326) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) ~[?:?]
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) ~[?:?]
        at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:91) ~[?:?]
        at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) ~[?:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ~[?:?]
        at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) ~[?:?]
        at org.elasticsearch.cloud.aws.InternalAwsS3Service.client(InternalAwsS3Service.java:70) ~[?:?]
        at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:345) ~[?:?]
        at org.elasticsearch.plugin.repository.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:71) ~[?:?]
        at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:383) ~[elasticsearch-6.0.0-alpha1-SNAPSHOT.jar:6.0.0-alpha1-SNAPSHOT]
        ... 14 more

[dadoonet]

Actually I did something wrong but still I'm getting this error.

Here is my mistake. I registered a client called my_config but I'm not using it when creating the repository.

With:

curl -X POST 'http://127.0.0.1:9200/_snapshot/backups?pretty=true&error_trace=true' -H 'Content-Type: application/json' -d '
{
   "type":"s3",
   "settings":{
      "client": "my_config",
      "bucket": "BUCKET",
      "region": "eu-west-1"
   }
}'

Everything works fine:

{
  "acknowledged" : true
}

So this appears only when no credentials are found. So when we switch to IAM Profiles instead.
In such case, AWS is trying to get the metadata from the instance by calling this URL 169.254.169.254:80.

I did not test but I believe that IAM Profile won't work.

[dadoonet]

Should be fixed by #23297