[CI] AutoFollowIT.testAutoFollowManyIndices test failure

[spinscale]

Could not reproduce this under Linux or osx. Link https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=oraclelinux/125/console

log snippet

  1> [2018-12-18T07:47:41,534][INFO ][o.e.x.c.AutoFollowIT     ] [testAutoFollowManyIndices] after test
FAILURE 15.9s J0 | AutoFollowIT.testAutoFollowManyIndices <<< FAILURES!
   > Throwable #1: java.lang.AssertionError:
   > Expected: <37>
   >      but: was <38>
   >    at __randomizedtesting.SeedInfo.seed([F02ACEB2EC9BD622:D818E1CF850A7355]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.xpack.ccr.AutoFollowIT.lambda$testAutoFollowManyIndices$6(AutoFollowIT.java:154)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:848)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:822)
   >    at org.elasticsearch.xpack.ccr.AutoFollowIT.testAutoFollowManyIndices(AutoFollowIT.java:151)
   >    at java.lang.Thread.run(Thread.java:748)
   >    Suppressed: java.lang.AssertionError:
   > Expected: <37>
   >      but: was <30>
   >            at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >            at org.elasticsearch.xpack.ccr.AutoFollowIT.lambda$testAutoFollowManyIndices$6(AutoFollowIT.java:154)
   >            at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
   >            ... 39 more
   >    Suppressed: java.lang.AssertionError:
   > Expected: <37>
   >      but: was <30>
   >            at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >            at org.elasticsearch.xpack.ccr.AutoFollowIT.lambda$testAutoFollowManyIndices$6(AutoFollowIT.java:154)
   >            at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
   >            ... 39 more
   >    Suppressed: java.lang.AssertionError:
   > Expected: <37>
   >      but: was <30>
   >            at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >            at org.elasticsearch.xpack.ccr.AutoFollowIT.lambda$testAutoFollowManyIndices$6(AutoFollowIT.java:154)
   >            at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:836)
   >            ... 39 more
   >    Suppressed: java.lang.AssertionError:

reproduction

./gradlew :x-pack:plugin:ccr:internalClusterTest \
  -Dtests.seed=F02ACEB2EC9BD622 \
  -Dtests.class=org.elasticsearch.xpack.ccr.AutoFollowIT \
  -Dtests.method="testAutoFollowManyIndices" \
  -Dtests.security.manager=true \
  -Dtests.locale=id \
  -Dtests.timezone=Africa/Monrovia \
  -Dcompiler.java=11 \
  -Druntime.java=8

Link to the plain text logfile is at https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=oraclelinux/125/consoleText

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

I've pushed this test fix: e4391af
I will close this issue if this test doen't fail in the coming days.

[gwbrown]

This test failed again today in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+multijob-unix-compatibility/os=debian/47/console

[martijnvg]

After taking a better look at the test and how the auto follow coordinator can work, I think that the underlying issue may be caused by a real issue in the auto follow coordinator.

The logs-does-not-count index should not get auto followed, but it does get auto followed. In the test an auto follow pattern is added, a number of leader indices get created that should get auto followed, the auto follow pattern gets removed, the logs-does-not-count gets created (and should not get auto followed), then the auto follow patterns get re-added and finally a number of leader indices get created that should get auto followed.

When the auto follow coordinator removes the auto follower, the auto follower itself doen't know it is deleted yet and may do another auto follow round before it really stops. I will open a PR to fix this.