Shared filesystem repository does not respect readonly setting

[DaveCTurner]

A user reports the following exception when restoring a snapshot from a readonly filesystem:

[2019-04-05T21:46:35,634][WARN ][o.e.s.RestoreService ] [REDACTED] [REDACTED:REDACTED] failed to restore snapshot 
org.elasticsearch.ElasticsearchException: failed to create blob container 
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:71) ~[elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getSnapshotIndexMetaData(BlobStoreRepository.java:591) ~[elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.snapshots.RestoreService.restoreSnapshot(RestoreService.java:210) [elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:91) [elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:49) [elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:124) [elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:211) [elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.1.jar:6.6.1] 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] 
at java.lang.Thread.run(Thread.java:834) [?:?] 
Caused by: java.nio.file.FileSystemException: REDACTED: Read-only file system 
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?] 
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?] 
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?] 
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:385) ~[?:?] 
at java.nio.file.Files.createDirectory(Files.java:689) ~[?:?] 
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:796) ~[?:?] 
at java.nio.file.Files.createDirectories(Files.java:782) ~[?:?] 
at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:88) ~[elasticsearch-6.6.1.jar:6.6.1] 
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:69) ~[elasticsearch-6.6.1.jar:6.6.1] 
... 11 more 

The following line looks to be bogus, because it checks for the readonly property of the node's settings rather than the repository metadata:

elasticsearch/server/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobStore.java

Line 45 in 0227260

this.readOnly = settings.getAsBoolean("readonly", false);

Relates #26909

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

The following line looks to be bogus, because it checks for the readonly property of the node's settings rather than the repository metadata:

Yea I think we should just pass down the readOnly setting as already parsed fororg.elasticsearch.repositories.blobstore.BlobStoreRepository#isReadOnly here.