-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Closed
Labels
:Distributed Coordination/Snapshot/RestoreAnything directly related to the `_snapshot/*` APIsAnything directly related to the `_snapshot/*` APIs>bug
Description
Performing these operations:
POST /test/doc/1
{"body": "foo"}
# make sure you have path.repo=/tmp
PUT /_snapshot/my_backup
{
"type": "fs",
"settings": {
"chunk_size": null,
"location": "/tmp/backups"
}
}
PUT /_snapshot/my_backup/snapshot-1?wait_for_completion=true
# the snapshot will show as successful
GET /_snapshot/my_backup/_all
POST /_snapshot/my_backup/snapshot-1/_restoreOn doing the restore, the index will be corrupt and fails with all kinds of:
[2017-09-29T15:32:51,804][WARN ][o.e.c.a.s.ShardStateAction] [zS7gqMd] [test][4] received shard failed for shard id [[test][4]], allocation id [1m3SAW23RiORYN3paf5zRg], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test][4]: Recovery failed on {zS7gqMd}{zS7gqMdkQgenxYl5YYMuPg}{fbrBGXZVSoOiHjueD2o3_Q}{127.0.0.1}{127.0.0.1:9300}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot-1/bBueY70xTW-w2r3yGoQrmA]]; nested: IndexShardRestoreFailedException[Failed to recover index]; nested: CorruptIndexException[verification failed (hardware problem?) : expected=j36fb4 actual=null footer=null writtenLength=0 expectedLength=197 (resource=name [segments_1], length [197], checksum [j36fb4], writtenBy [6.2.0]) (resource=VerifyingIndexOutput(segments_1))]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [test][4]: Recovery failed on {zS7gqMd}{zS7gqMdkQgenxYl5YYMuPg}{fbrBGXZVSoOiHjueD2o3_Q}{127.0.0.1}{127.0.0.1:9300}
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1710) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:305) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:413) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [snapshot-1/bBueY70xTW-w2r3yGoQrmA]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:844) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: Failed to recover index
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1524) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:842) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
... 4 more
Caused by: org.apache.lucene.index.CorruptIndexException: verification failed (hardware problem?) : expected=j36fb4 actual=null footer=null writtenLength=0 expectedLength=197 (resource=name [segments_1], length [197], checksum [j36fb4], writtenBy [6.2.0]) (resource=VerifyingIndexOutput(segments_1))
at org.elasticsearch.index.store.Store$LuceneVerifyingIndexOutput.verify(Store.java:1124) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.store.Store.verify(Store.java:464) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restoreFile(BlobStoreRepository.java:1586) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1521) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:842) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:408) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:240) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:263) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:238) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1317) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1706) ~[elasticsearch-6.0.0-alpha1.jar:6.0.0-alpha1]
... 4 more
Metadata
Metadata
Assignees
Labels
:Distributed Coordination/Snapshot/RestoreAnything directly related to the `_snapshot/*` APIsAnything directly related to the `_snapshot/*` APIs>bug