[CI] MasterDisruptionIT.testVerifyApiBlocksDuringPartition fails in checkIndex

[droberts195]

MasterDisruptionIT.testVerifyApiBlocksDuringPartition failed on the 6.x branch on macOS: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-darwin-compatibility/398/console

This doesn't reproduce locally for me with:

gradle :core:integTest \
  -Dtests.seed=46C91E745D178048 \
  -Dtests.class=org.elasticsearch.discovery.MasterDisruptionIT \
  -Dtests.method="testVerifyApiBlocksDuringPartition" \
  -Dtests.security.manager=true \
  -Dtests.locale=fr-CA \
  -Dtests.timezone=America/Swift_Current

It actually failed after the test in the cleanup checks of ESTestCase:

10:41:51 FAILURE 22.9s J2 | MasterDisruptionIT.testVerifyApiBlocksDuringPartition <<< FAILURES!
10:41:51    > Throwable #1: java.lang.AssertionError: at least one shard failed CheckIndex
10:41:51    > 	at __randomizedtesting.SeedInfo.seed([46C91E745D178048:A671712025C4F3C2]:0)
10:41:51    > 	at org.elasticsearch.test.ESTestCase.ensureCheckIndexPassed(ESTestCase.java:434)
10:41:51    > 	at org.elasticsearch.test.ESTestCase.after(ESTestCase.java:301)
10:41:51    > 	at java.lang.Thread.run(Thread.java:748)

This implies MockFSDirectoryService's index check failed, and this is the exception that caused that:

10:41:49   1> [2017-12-08T04:40:46,912][WARN ][o.e.t.s.MockFSIndexStore$Listener] [node_t1] [test][0] failed to check index
10:41:49   1> java.lang.IllegalStateException: IndexWriter is still open on shard [test][0]
10:41:49   1> 	at org.elasticsearch.test.store.MockFSDirectoryService.checkIndex(MockFSDirectoryService.java:141) ~[framework-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
10:41:49   1> 	at org.elasticsearch.test.store.MockFSIndexStore$Listener.afterIndexShardClosed(MockFSIndexStore.java:99) ~[framework-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
10:41:49   1> 	at org.elasticsearch.index.CompositeIndexEventListener.afterIndexShardClosed(CompositeIndexEventListener.java:111) ~[main/:?]
10:41:49   1> 	at org.elasticsearch.index.IndexService.closeShard(IndexService.java:439) ~[main/:?]
10:41:49   1> 	at org.elasticsearch.index.IndexService.removeShard(IndexService.java:414) ~[main/:?]
10:41:49   1> 	at org.elasticsearch.index.IndexService.close(IndexService.java:274) ~[main/:?]
10:41:49   1> 	at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:552) ~[main/:?]
10:41:49   1> 	at org.elasticsearch.indices.cluster.IndicesClusterStateService.removeUnallocatedIndices(IndicesClusterStateService.java:366) ~[main/:?]
10:41:49   1> 	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:221) ~[main/:?]
10:41:49   1> 	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:498) ~[main/:?]
10:41:49   1> 	at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_141]
10:41:49   1> 	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:495) [main/:?]
10:41:49   1> 	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:482) [main/:?]
10:41:49   1> 	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) [main/:?]
10:41:49   1> 	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?]
10:41:49   1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566) [main/:?]
10:41:49   1> 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [main/:?]
10:41:49   1> 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [main/:?]
10:41:49   1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
10:41:49   1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
10:41:49   1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
10:41:49   1> Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /private/var/lib/jenkins/workspace/elastic+elasticsearch+6.x+multijob-darwin-compatibility/core/build/testrun/integTest/J2/temp/org.elasticsearch.discovery.MasterDisruptionIT_46C91E745D178048-001/tempDir-003/data/nodes/1/indices/gXmY0hObRUS-fnf_uDDP0w/0/index/write.lock
10:41:49   1> 	at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:139) ~[lucene-core-7.2.0-snapshot-7deca62.jar:7.2.0-snapshot-7deca62 7deca62501ec7484ea54d292fe0131c78384e95f - jpountz - 2017-12-01 12:38:44]
10:41:49   1> 	at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.2.0-snapshot-7deca62.jar:7.2.0-snapshot-7deca62 7deca62501ec7484ea54d292fe0131c78384e95f - jpountz - 2017-12-01 12:38:44]
10:41:49   1> 	at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.2.0-snapshot-7deca62.jar:7.2.0-snapshot-7deca62 7deca62501ec7484ea54d292fe0131c78384e95f - jpountz - 2017-12-01 12:38:44]
10:41:49   1> 	at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:104) ~[lucene-core-7.2.0-snapshot-7deca62.jar:7.2.0-snapshot-7deca62 7deca62501ec7484ea54d292fe0131c78384e95f - jpountz - 2017-12-01 12:38:44]
10:41:49   1> 	at org.apache.lucene.store.MockDirectoryWrapper.obtainLock(MockDirectoryWrapper.java:1049) ~[lucene-test-framework-7.2.0-snapshot-7deca62.jar:7.2.0-snapshot-7deca62 7deca62501ec7484ea54d292fe0131c78384e95f - jpountz - 2017-12-01 12:38:44]
10:41:49   1> 	at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:104) ~[lucene-core-7.2.0-snapshot-7deca62.jar:7.2.0-snapshot-7deca62 7deca62501ec7484ea54d292fe0131c78384e95f - jpountz - 2017-12-01 12:38:44]
10:41:49   1> 	at org.apache.lucene.index.CheckIndex.<init>(CheckIndex.java:403) ~[lucene-core-7.2.0-snapshot-7deca62.jar:7.2.0-snapshot-7deca62 7deca62501ec7484ea54d292fe0131c78384e95f - jpountz - 2017-12-01 12:38:44]
10:41:49   1> 	at org.elasticsearch.test.store.MockFSDirectoryService.checkIndex(MockFSDirectoryService.java:122) ~[framework-6.2.0-SNAPSHOT.jar:6.2.0-SNAPSHOT]
10:41:49   1> 	... 20 more

[jpountz]

I double checked that we make sure to close the engine, which in turn closes the index writer, before calling the afterIndexShardClosed listeners. So there is something fishy happening. @simonw Pinging you since you are more familiar with the engine than I am.

[s1monw]

I think this will be fixed by #27768