Skip to content

Creating a repository with chunk_size: null causes snapshots to be corrupted #26843

@dakrone

Description

@dakrone

Performing these operations:

POST /test/doc/1
{"body": "foo"}

# make sure you have path.repo=/tmp
PUT /_snapshot/my_backup
{
  "type": "fs",
  "settings": {
    "chunk_size": null,
    "location": "/tmp/backups"
  }
}

PUT /_snapshot/my_backup/snapshot-1?wait_for_completion=true

# the snapshot will show as successful
GET /_snapshot/my_backup/_all

POST /_snapshot/my_backup/snapshot-1/_restore

On doing the restore, the index will be corrupt and fails with all kinds of:

[2017-09-29T15:32:51,804][WARN ][o.e.c.a.s.ShardStateAction] [zS7gqMd] [test][4] received shard failed for shard id [[test][4]], allocation id [1m3SAW23RiORYN3paf5zRg], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test][4]: Recovery failed on {zS7gqMd}{zS7gqMdkQgenxYl5YYMuPg}{fbrBGXZVSoOiHjueD2o3_Q}{127.0.0.1}{127.0.0.1:9300}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot-1/bBueY70xTW-w2r3yGoQrmA]]; nested: IndexShardRestoreFailedException[Failed to recover index]; nested: CorruptIndexException[verification failed (hardware problem?) : expected=j36fb4 actual=null footer=null writtenLength=0 expectedLength=197 (resource=name [segments_1], length [197], checksum [j36fb4], writtenBy [6.2.0]) (resource=VerifyingIndexOutput(segments_1))]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [test][4]: Recovery failed on {zS7gqMd}{zS7gqMdkQgenxYl5YYMuPg}{fbrBGXZVSoOiHjueD2o3_Q}{127.0.0.1}{127.0.0.1:9300}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1710) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:305) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:413) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [snapshot-1/bBueY70xTW-w2r3yGoQrmA]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:844) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: Failed to recover index
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1524) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:842) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more
Caused by: org.apache.lucene.index.CorruptIndexException: verification failed (hardware problem?) : expected=j36fb4 actual=null footer=null writtenLength=0 expectedLength=197 (resource=name [segments_1], length [197], checksum [j36fb4], writtenBy [6.2.0]) (resource=VerifyingIndexOutput(segments_1))
	at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.verify(Store.java:1124) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.store.Store.verify(Store.java:464) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restoreFile(BlobStoreRepository.java:1586) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1521) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:842) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
	... 4 more

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions