IndexFollowingIT.testDeleteLeaderIndex fails in CI

[romseygeek]

First failure today, does not reproduce:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+internalClusterTest/1693/console

java.lang.AssertionError: 
Expected: "no such index [index1]"
     but: was null
	at __randomizedtesting.SeedInfo.seed([851893BC292287A4:CB6400D08EF018D7]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.xpack.ccr.IndexFollowingIT.lambda$testDeleteLeaderIndex$22(IndexFollowingIT.java:686)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:868)
	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:842)
	at org.elasticsearch.xpack.ccr.IndexFollowingIT.testDeleteLeaderIndex(IndexFollowingIT.java:678)

REPRODUCE WITH: ./gradlew :x-pack:plugin:ccr:internalClusterTest \
  -Dtests.seed=851893BC292287A4 \
  -Dtests.class=org.elasticsearch.xpack.ccr.IndexFollowingIT \
  -Dtests.method="testDeleteLeaderIndex" \
  -Dtests.security.manager=true \
  -Dtests.locale=zh-HK \
  -Dtests.timezone=Pacific/Kwajalein \
  -Dcompiler.java=11 \
  -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

The actual cause of the failure is that instead of index following failing with an index not found exception, index following fails with a NPE:

[2019-02-22T19:10:24,144][WARN ][o.e.x.c.a.ShardFollowNodeTask] [follower1] shard follow task encounter non-retryable error
  1> org.elasticsearch.transport.RemoteTransportException: [leaderd2][127.0.0.1:34037][indices:data/read/xpack/ccr/shard_changes]
  1> Caused by: org.elasticsearch.transport.RemoteTransportException: [leaderd2][127.0.0.1:34037][indices:data/read/xpack/ccr/shard_changes[s]]
  1> Caused by: java.lang.NullPointerException
  1>    at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.globalCheckpointAdvancementFailure(ShardChangesAction.java:421) ~[main/:?]
  1>    at org.elasticsearch.xpack.ccr.action.ShardChangesAction$TransportAction.lambda$asyncShardOperation$0(ShardChangesAction.java:386) ~[main/:?]
  1>    at org.elasticsearch.index.shard.GlobalCheckpointListeners.notifyListener(GlobalCheckpointListeners.java:229) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT
]
  1>    at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$add$2(GlobalCheckpointListeners.java:143) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
  1>    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSH
OT]
  1>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
  1>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
  1>    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
  1>    Suppressed: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: timeout_exception: 10ms
  1>            at org.elasticsearch.index.shard.GlobalCheckpointListeners.lambda$add$3(GlobalCheckpointListeners.java:141) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SN
APSHOT]
  1>            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202]
  1>            at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202]
  1>            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202]
  1>            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202]
  1>            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202]
  1>            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202]
  1>            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

This fix is simple. In ShardChangesAction.TransportAction#globalCheckpointAdvancementFailure(...) method we need to guard against the fact that the IndexMetaData instance for the leader index is missing.