Shrink upgraded index (created on 5.x) fails in 6.x

[gmoskovicz]

Elasticsearch version (bin/elasticsearch --version): 6.0/ Any 6.x version

Plugins installed: []

JVM version (java -version): 1.8 JDK

OS version (uname -a if on a Unix-like system): Any

Description of the problem including expected versus actual behavior:

Steps to reproduce:

[1] Create in 5.6.x a test index

curl -XPOST localhost:9200/test/test -d '{ "test":true }' 

[2] Upgrade 5.6.x to 6.x

[3] Start Elasticsearch

[4] Run the shrinking

PUT test/_settings
{
  "settings": {
    "index.blocks.write": true 
  }
}

POST test/_shrink/test_shrink
{
  "settings": {
    "index.number_of_replicas": 1,
    "index.number_of_shards": 1, 
    "index.codec": "best_compression" 
  }
}

[5] Check the cluster health

{
  "cluster_name": "elasticsearch",
  "status": "red",     <----------------- IT's red one
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 10,
  "active_shards": 10,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 11,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 47.61904761904761
}

[7] Check the RED indices

test_shrink                     0 p UNASSIGNED                        
test_shrink                     0 r UNASSIGNED                        

Logs show the following

2018-01-03T10:22:23,627][WARN ][o.e.c.a.s.ShardStateAction] [-VTajZI] [test_shrink][0] received shard failed for shard id [[test_shrink][0]], allocation id [hYCRkpmGRRC7CZCQ_As67g], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test_shrink][0]: Recovery failed on {-VTajZI}{-VTajZIrRT66GVxLwD4X6w}{qW38ZFvvRFCLi0miV_YWMA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17179869184, ml.max_open_jobs=20, ml.enabled=true}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalArgumentException[Cannot use addIndexes(Directory) with indexes that have been created by a different Lucene version. The current index was generated by Lucene 7 while one of the directories contains an index that was generated with Lucene 6]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [test_shrink][0]: Recovery failed on {-VTajZI}{-VTajZIrRT66GVxLwD4X6w}{qW38ZFvvRFCLi0miV_YWMA}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=17179869184, ml.max_open_jobs=20, ml.enabled=true}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2077) [elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.IndexShard$$Lambda$2882/401965331.run(Unknown Source) [elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.1.jar:6.1.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:334) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:122) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1565) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2072) ~[elasticsearch-6.1.1.jar:6.1.1]
	... 5 more
Caused by: java.lang.IllegalArgumentException: Cannot use addIndexes(Directory) with indexes that have been created by a different Lucene version. The current index was generated by Lucene 7 while one of the directories contains an index that was generated with Lucene 6
	at org.apache.lucene.index.IndexWriter.addIndexes(IndexWriter.java:2830) ~[lucene-core-7.1.0.jar:7.1.0 84c90ad2c0218156c840e19a64d72b8a38550659 - ubuntu - 2017-10-13 16:12:42]
	at org.elasticsearch.index.shard.StoreRecovery.addIndices(StoreRecovery.java:161) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromLocalShards$3(StoreRecovery.java:130) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.StoreRecovery$$Lambda$2886/1712721504.run(Unknown Source) ~[?:?]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:292) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromLocalShards(StoreRecovery.java:122) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.IndexShard.recoverFromLocalShards(IndexShard.java:1565) ~[elasticsearch-6.1.1.jar:6.1.1]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2072) ~[elasticsearch-6.1.1.jar:6.1.1]
	... 5 more
[2018-01-03T10:22:23,634][INFO ][o.e.c.r.a.AllocationService] [-VTajZI] Cluster health status changed from [YELLOW] to [RED] (reason: [shards failed [[test_shrink][0]] ...]).

The issue seems to be:

Caused by: java.lang.IllegalArgumentException: Cannot use addIndexes(Directory) with indexes that have been created by a different Lucene version. The current index was generated by Lucene 7 while one of the directories contains an index that was generated with Lucene 6

CC @s1monw here, i think that this might be caused by #22469 ?

[jpountz]

I don't think this is due to #22469. Lucene complains that the creation versions differ because we changed the way norms are encoded in Lucene 7, so you can't add Lucene 6 files to a Lucene 7 index. We should fix shrinking to also carry over the Lucene creation version, in addition to the Elasticsearch creation version.