FS-based repository does not honour readonly setting when restoring from a broken snapshot

[ywelsch]

The FS-based repository does not honour the readonly setting when restoring from a broken snapshot (in fact, restoring a snapshot should not require any writes at all).

A write can currently happen, however, if the repository has missing index folders.

Reproduction scenario:

1. Create snapshot of an index
2. Remove the folder in the repository under indices
3. Try to restore the snapshot. This will try to recreate said folder.

Stack trace when using a read-only filesystem:

[2016-11-11T20:32:22,830][WARN ][o.e.s.RestoreService     ] [node_sm0] [test-repo:test-snap] failed to restore snapshot
org.elasticsearch.ElasticsearchException: failed to create blob container
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:67) ~[main/:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotMetaData(BlobStoreRepository.java:599) ~[main/:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.getSnapshotMetaData(BlobStoreRepository.java:554) ~[main/:?]
	at org.elasticsearch.snapshots.RestoreService.restoreSnapshot(RestoreService.java:187) ~[main/:?]
	at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:89) ~[main/:?]
	at org.elasticsearch.action.admin.cluster.snapshots.restore.TransportRestoreSnapshotAction.masterOperation(TransportRestoreSnapshotAction.java:49) ~[main/:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) ~[main/:?]
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$3.doRun(TransportMasterNodeAction.java:171) ~[main/:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:520) ~[main/:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[main/:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_60]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_60]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_60]
Caused by: java.nio.file.AccessDeniedException: /private/var/folders/68/3gzf12zs4qb0q_gfjw5lx1fm0000gn/T/org.elasticsearch.snapshots.SharedClusterSnapshotRestoreIT_73B55FFF46128756-001/tempDir-002/repos/prkihNFaMA/indices/Odv1GtZFSlaFEOwTZWQcdQ
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
	at org.apache.lucene.mockfile.FilterFileSystemProvider.createDirectory(FilterFileSystemProvider.java:132) ~[lucene-test-framework-6.3.0-snapshot-a66a445.jar:6.3.0-snapshot-a66a445 a66a44513ee8191e25b477372094bfa846450316 - jpountz - 2016-11-03 16:32:47]
	at org.apache.lucene.mockfile.FilterFileSystemProvider.createDirectory(FilterFileSystemProvider.java:132) ~[lucene-test-framework-6.3.0-snapshot-a66a445.jar:6.3.0-snapshot-a66a445 a66a44513ee8191e25b477372094bfa846450316 - jpountz - 2016-11-03 16:32:47]
	at org.apache.lucene.mockfile.FilterFileSystemProvider.createDirectory(FilterFileSystemProvider.java:132) ~[lucene-test-framework-6.3.0-snapshot-a66a445.jar:6.3.0-snapshot-a66a445 a66a44513ee8191e25b477372094bfa846450316 - jpountz - 2016-11-03 16:32:47]
	at org.apache.lucene.mockfile.FilterFileSystemProvider.createDirectory(FilterFileSystemProvider.java:132) ~[lucene-test-framework-6.3.0-snapshot-a66a445.jar:6.3.0-snapshot-a66a445 a66a44513ee8191e25b477372094bfa846450316 - jpountz - 2016-11-03 16:32:47]
	at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_60]
	at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_60]
	at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_60]
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:83) ~[main/:?]
	at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:65) ~[main/:?]
	... 12 more

Line at fault:

elasticsearch/core/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobStore.java

Line 65 in 44ac5d0

return new FsBlobContainer(this, path, buildAndCreate(path));

Just getting a BlobContainer for reading does an mkdirs on the root directory for that container.

[liketic]

@ywelsch Can I work on this?

[ywelsch]

@liketic sure