org.elasticsearch.action.admin.indices.create.ShrinkIndexIT#testShrinkIndexPrimaryTerm() fails in CI

[danielmitterdorfer]

• Test failure link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu/1172/consoleFull
• Failure output:

ERROR   1.99s J0 | ShrinkIndexIT.testShrinkIndexPrimaryTerm <<< FAILURES!
   > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2450, name=elasticsearch[node_s0][refresh][T#1], state=RUNNABLE, group=TGRP-ShrinkIndexIT]
   > 	at __randomizedtesting.SeedInfo.seed([5EE4DE9941E6916F:7468EB005FBE8422]:0)
   > Caused by: java.lang.AssertionError: segState.delGen=1 vs this.gen=1
   > 	at __randomizedtesting.SeedInfo.seed([5EE4DE9941E6916F]:0)
   > 	at org.apache.lucene.index.FrozenBufferedUpdates.applyTermDeletes(FrozenBufferedUpdates.java:722)
   > 	at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:435)
   > 	at org.apache.lucene.index.FrozenBufferedUpdates.apply(FrozenBufferedUpdates.java:323)
   > 	at org.apache.lucene.index.DocumentsWriter$ResolveUpdatesEvent.process(DocumentsWriter.java:719)
   > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5061)
   > 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5049)
   > 	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:466)
   > 	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293)
   > 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268)
   > 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258)
   > 	at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104)
   > 	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
   > 	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156)
   > 	at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
   > 	at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
   > 	at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253)
   > 	at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1118)
   > 	at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:787)
   > 	at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:683)
   > 	at org.elasticsearch.index.IndexService.access$400(IndexService.java:91)
   > 	at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:826)
   > 	at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:736)
   > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   > 	at java.lang.Thread.run(Thread.java:748)

• Reproduction line: gradle :core:integTest -Dtests.seed=5EE4DE9941E6916F -Dtests.class=org.elasticsearch.action.admin.indices.create.ShrinkIndexIT -Dtests.method="testShrinkIndexPrimaryTerm" -Dtests.security.manager=true -Dtests.locale=es-BO -Dtests.timezone=America/Dawson_Creek
• Frequency: failed only once so far in CI; does not reproduce locally

This may be related to #25122 but hard to tell for me based on the output.

[jasontedor]

I don't think this one has anything to do with #25122, nor anything to do with #25321 and #25356. Rather, this is manifesting in the Lucene layer so I would like to ask @jpountz to triage this one?

[jpountz]

I've been digging this bug and suspect it is due to bugs we had with concurrent updates introduced in https://issues.apache.org/jira/browse/LUCENE-7868. These bugs have been fixed and we upgraded to more recent snapshots so we should not be seeing this error anymore.

[jasontedor]

Thank you @jpountz!