[CI] IndexAuditUpgradeIT.testDocsAuditedInMixedCluster

[romseygeek]

A number of failures on this one over the weekend:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=oraclelinux/63/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=debian/63/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=fedora/61/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=ubuntu&&virtual/61/console

REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.2.5-SNAPSHOT#mixedClusterTestRunner \
  -Dtests.seed=5B3EE62A13DCC0EB \
  -Dtests.class=org.elasticsearch.upgrades.IndexAuditUpgradeIT \
  -Dtests.method="testDocsAuditedInMixedCluster" \
  -Dtests.security.manager=true \
  -Dtests.locale=es-MX \
  -Dtests.timezone=America/Chicago

java.lang.AssertionError: 
Expected: is <2>
     but: was <3>
	at __randomizedtesting.SeedInfo.seed([5B3EE62A13DCC0EB:5ACAB73AA5CC61FA]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
	at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInMixedCluster$1(IndexAuditUpgradeIT.java:62)

[elasticmachine]

Pinging @elastic/es-security

[tlrx]

Another instance: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+bwc-tests/887

00:00:43 IGNOR/A 0.04s | IndexAuditUpgradeIT.testDocsAuditedInOldCluster
00:00:43 > Assumption #1: only runs against old cluster
00:00:43 1> [2018-05-22T10:00:32,809][INFO ][o.e.u.IndexAuditUpgradeIT] [testDocsAuditedInUpgradedCluster]: before test
00:00:43 1> [2018-05-22T10:00:32,825][ERROR][o.e.u.IndexAuditUpgradeIT] get settings response {.security_audit_log-2018.05.22={settings={index.version.created=5060199}}, .security_audit_log-2018.05.21={settings={index.version.created=5060199}}}
00:00:43 1> [2018-05-22T10:00:43,225][INFO ][o.e.u.IndexAuditUpgradeIT] [testDocsAuditedInUpgradedCluster]: after test
00:00:43 2> REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v5.6.1#upgradedClusterTestRunner -Dtests.seed=D5E304FD8B648875 -Dtests.class=org.elasticsearch.upgrades.IndexAuditUpgradeIT -Dtests.method="testDocsAuditedInUpgradedCluster" -Dtests.security.manager=true -Dtests.locale=ru -Dtests.timezone=AET
00:00:43 FAILURE 10.4s | IndexAuditUpgradeIT.testDocsAuditedInUpgradedCluster <<< FAILURES!
00:00:43 > Throwable #1: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at __randomizedtesting.SeedInfo.seed([D5E304FD8B648875:A8875756F34BCEDC]:0)
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:767)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:741)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.testDocsAuditedInUpgradedCluster(IndexAuditUpgradeIT.java:89)
00:00:43 > at java.lang.Thread.run(Thread.java:748)
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=443, key=mixed-node-0}]
00:00:43 > Expected: is <2>
00:00:43 > but: was <1>
00:00:43 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:43 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:91)
00:00:43 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
00:00:43 > ... 39 more
00:00:43 1> [2018-05-22T10:00:43,242][INFO ][o.e.u.IndexAuditUpgradeIT] [testDocsAuditedInMixedCluster]: before test
00:00:43 1> [2018-05-22T10:00:43,263][INFO ][o.e.u.IndexAuditUpgradeIT] [testDocsAuditedInMixedCluster]: after test
00:00:43 IGNOR/A 0.03s | IndexAuditUpgradeIT.testDocsAuditedInMixedCluster
00:00:43 > Assumption #1: only runs against mixed cluster
00:00:43 2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.x+bwc-tests/x-pack/qa/rolling-upgrade/with-system-key/build/testrun/v5.6.1#upgradedClusterTestRunner/J0/temp/org.elasticsearch.upgrades.IndexAuditUpgradeIT_D5E304FD8B648875-001
00:00:43 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1735, maxMBSortInHeap=5.935387284784882, sim=RandomSimilarity(queryNorm=true): {}, locale=ru, timezone=AET
00:00:43 2> NOTE: Linux 4.4.0-1057-aws amd64/Oracle Corporation 1.8.0_162 (64-bit)/cpus=16,threads=1,free=371099336,total=514850816
00:00:43 2> NOTE: All tests run in this JVM: [WatchBackwardsCompatibilityIT, TokenBackwardsCompatibilityIT, IndexAuditUpgradeIT]
00:00:43 Completed [3/4] in 10.54s, 3 tests, 1 failure, 2 skipped <<< FAILURES!
00:00:43
00:00:48 Suite: org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT
00:00:48 IGNOR/A 0.06s | UpgradeClusterClientYamlTestSuiteIT.test {p0=upgraded_cluster/30_ml_jobs_crud/Test model memory limit is updated}
00:00:48 > Assumption #1: [upgraded_cluster/30_ml_jobs_crud/Test model memory limit is updated] skipped, reason: blacklisted
00:00:48 Completed [4/4] in 4.81s, 10 tests, 1 skipped

[cbuescher]

Another one here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.3+multijob-unix-compatibility/os=amazon/93/console

REPRODUCE WITH: ./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.2.5-SNAPSHOT#upgradedClusterTestRunner \
  -Dtests.seed=7BF5E38B468E12ED \
  -Dtests.class=org.elasticsearch.upgrades.IndexAuditUpgradeIT \
  -Dtests.method="testDocsAuditedInUpgradedCluster" \
  -Dtests.security.manager=true \
  -Dtests.locale=da-DK \
  -Dtests.timezone=America/St_Lucia

FAILURE 10.7s | IndexAuditUpgradeIT.testDocsAuditedInUpgradedCluster <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: Found node buckets [{doc_count=3419, key=mixed-node-0}, {doc_count=3102, key=node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 	at __randomizedtesting.SeedInfo.seed([7BF5E38B468E12ED:691B0203EA15444]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 	at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:767)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:741)
   > 	at org.elasticsearch.upgrades.IndexAuditUpgradeIT.testDocsAuditedInUpgradedCluster(IndexAuditUpgradeIT.java:90)
   > 	at java.lang.Thread.run(Thread.java:748)
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>
   > 		at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:127)
   > 		at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testDocsAuditedInUpgradedCluster$2(IndexAuditUpgradeIT.java:92)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:755)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=3102, key=node-0}, {doc_count=3029, key=mixed-node-0}, {doc_count=1389, key=node-1}]
   > Expected: is <4>
   >      but: was <3>

[nik9000]

I bumped into this when I was working on switching from two to three nodes in the rolling restart - it has to do with when a new master is elected with the new version. I believe I'll have fixed this when I backport #31112.

[nik9000]

I'm fairly sure I fixed this as part of b88d3d3.