[CI] MixedClusterClientYamlTestSuiteIT test {p0=field_caps/10_basic/Get object and nested field caps} failing

[martijnvg]

Build scan:
https://gradle-enterprise.elastic.co/s/lgvrzlewzdu3e/tests/:qa:mixed-cluster:v7.7.0%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=field_caps%2F10_basic%2FGet%20object%20and%20nested%20field%20caps%7D

Reproduction line:
./gradlew ':qa:mixed-cluster:v7.7.0#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=field_caps/10_basic/Get object and nested field caps}" -Dtests.seed=E59B64A1F501FD5 -Dtests.bwc=true -Dtests.locale=es-AR -Dtests.timezone=IST -Druntime.java=8

Applicable branches:
7.16

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT&tests.test=test%20%7Bp0%3Dfield_caps/10_basic/Get%20object%20and%20nested%20field%20caps%7D

Failure excerpt:

java.lang.AssertionError: Failure at [field_caps/10_basic:216]: field [fields.object.object.indices] is null

  at __randomizedtesting.SeedInfo.seed([E59B64A1F501FD5:860D8990B1AC722D]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:482)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:455)
  at sun.reflect.GeneratedMethodAccessor21.invoke(null:-1)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at java.lang.Thread.run(Thread.java:748)

  Caused by: java.lang.AssertionError: field [fields.object.object.indices] is null

    at org.junit.Assert.fail(Assert.java:88)
    at org.junit.Assert.assertTrue(Assert.java:41)
    at org.junit.Assert.assertNotNull(Assert.java:712)
    at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:78)
    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:65)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:471)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:455)
    at sun.reflect.GeneratedMethodAccessor21.invoke(null:-1)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
    at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
    at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
    at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
    at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[cbuescher]

@martijnvg I was looking if this error ever occured in our main CI tests but couldn't find anything similar so far, you saw this in #80330, right? In terms of reproduction: that PR was running bwc pn 7.16 against 7.7 if I understand the repro line correctly?

[cbuescher]

I found one similar looking error in a CI run from Oct 19th now, but not many more:
https://gradle-enterprise.elastic.co/s/ncx576tsubet6

./gradlew ':qa:mixed-cluster:v7.5.2#mixedClusterTest' \
  -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" \
  -Dtests.method="test {p0=field_caps/10_basic/Get object and nested field caps}" \
  -Dtests.seed=1DC3F29F235E175B \
  -Dtests.bwc=true \
  -Dtests.locale=zh \
  -Dtests.timezone=Canada/Saskatchewan \
  -Druntime.java=8

java.lang.AssertionError: Failure at [field_caps/10_basic:216]: field [fields.object.object.indices] is null
	at __randomizedtesting.SeedInfo.seed([1DC3F29F235E175B:9597CD458DA27AA3]:0)
	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:462)

 1> [2021-10-18T16:05:38,569][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "indices" : [
  1>         "test1",
  1>         "test2",
  1>         "test3"
  1>       ],
  1>       "fields" : {
  1>         "object.nested2" : {
  1>           "float" : {
  1>             "type" : "float",
  1>             "metadata_field" : false,
  1>             "searchable" : true,
  1>             "aggregatable" : false,
  1>             "non_aggregatable_indices" : [
  1>               "test1"
  1>             ]
  1>           }
  1>         },
  1>         "level1.level2.leaf1" : {
  1>           "text" : {
  1>             "type" : "text",
  1>             "metadata_field" : false,
  1>             "searchable" : false,
  1>             "aggregatable" : false
  1>           }
  1>         },
  1>         "level1.level2" : {
  1>           "object" : {
  1>             "type" : "object",
  1>             "metadata_field" : false,
  1>             "searchable" : false,
  1>             "aggregatable" : false
  1>           }
  1>         },
  1>         "object.nested1" : {
  1>           "text" : {
  1>             "type" : "text",
  1>             "metadata_field" : false,
  1>             "searchable" : false,
  1>             "aggregatable" : false,
  1>             "non_searchable_indices" : [
  1>               "test1"
  1>             ]
  1>           }
  1>         },
  1>         "level1" : {
  1>           "nested" : {
  1>             "type" : "nested",
  1>             "metadata_field" : false,
  1>             "searchable" : false,
  1>             "aggregatable" : false
  1>           }
  1>         },
  1>         "object" : {
  1>           "object" : {
  1>             "type" : "object",
  1>             "metadata_field" : false,
  1>             "searchable" : false,
  1>             "aggregatable" : false
  1>           }
  1>         }
  1>       }
  1>     }
  1>   }
  1> }]

[cbuescher]

Could not reproduce this locally yet despite many runs with different seeds, I will also try running the same seed multiple times.
The expected output of that particular "_field_caps" call in the "fields.object.object..." branch should be:

"object": {
	"nested": {
		"type": "nested",
		"searchable": false,
		"aggregatable": false,
		"indices": [
			"test3"
		]
	},
	"object": {
		"type": "object",
		"searchable": false,
		"aggregatable": false,
		"indices": [
			"test1",
			"test2"
		]
	}
}

The "indices" section should only be left out if either all indices where mapped the same way or if "test3" didn't exist, but its clearly there and should be mapped differently.

[cbuescher]

I'm looking at trying to reproduce this with some unexpected ordering in which the FieldCapabilitiesIndexResponse arrive and get merged, but so far I haven't succeeded or seen how in the code this could lead to a missing "indices" entry in that particular part of the output.

[cbuescher]

Maybe not directly related, but in a failure from Nov 7th. I see an index shard failure in this tests method, also in a mixed cluster scenario:

java.lang.AssertionError: Failure at [field_caps/10_basic:209]: Expected a list containing
0: expected “test1” but was “test2”
1: expected “test2” but was <missing>

at __randomizedtesting.SeedInfo.seed([8F52471DA528A074:70678C70BD4CD8C]:0)


Caused by: java.lang.AssertionError: Expected a list containing
0: expected “test1” but was “test2”
1: expected “test2” but was <missing>

at io.github.nik9000.mapmatcher.MapMatcher.assertMap(MapMatcher.java:79)



[2021-11-07T14:00:26,927][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=field_caps/10_basic/Get object and nested field caps] before test
[2021-11-07T14:00:27,100][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  "stash" : {
    "body" : {
      "indices" : [
        "test2",
        "test3"
      ],
      "fields" : {
        "object.nested2" : {
          "keyword" : {
            "type" : "keyword",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : false,
            "indices" : [
              "test3"
            ]
          },
          "float" : {
            "type" : "float",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : true,
            "indices" : [
              "test2"
            ]
          }
        },
        "level1.level2.leaf1" : {
          "text" : {
            "type" : "text",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false
          }
        },
        "level1.level2" : {
          "object" : {
            "type" : "object",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false
          }
        },
        "object.nested1" : {
          "text" : {
            "type" : "text",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : false,
            "indices" : [
              "test2"
            ]
          },
          "long" : {
            "type" : "long",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : true,
            "indices" : [
              "test3"
            ]
          }
        },
        "level1" : {
          "nested" : {
            "type" : "nested",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false,
            "indices" : [
              "test2"
            ]
          },
          "object" : {
            "type" : "object",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false,
            "indices" : [
              "test3"
            ]
          }
        },
        "object" : {
          "nested" : {
            "type" : "nested",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false,
            "indices" : [
              "test3"
            ]
          },
          "object" : {
            "type" : "object",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false,
            "indices" : [
              "test2"
            ]
          }
        }
      },
      "failed_indices" : 1,
      "failures" : [
        {
          "indices" : [
            "test1"
          ],
          "failure" : {
            "error" : {
              "root_cause" : [
                {
                  "type" : "illegal_index_shard_state_exception",
                  "reason" : "CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED]",
                  "index_uuid" : "6ZsTD9fJSDOJ8WiRURGf2A",
                  "shard" : "0",
                  "index" : "test1",
                  "suppressed" : [
                    {
                      "type" : "index_not_found_exception",
                      "reason" : "no such index [test1]",
                      "index_uuid" : "6ZsTD9fJSDOJ8WiRURGf2A",
                      "index" : "test1"
                    }
                  ]
                }
              ],
              "type" : "illegal_index_shard_state_exception",
              "reason" : "CurrentState[RECOVERING] operations only allowed when shard state is one of [POST_RECOVERY, STARTED]",
              "index_uuid" : "6ZsTD9fJSDOJ8WiRURGf2A",
              "shard" : "0",
              "index" : "test1",
              "suppressed" : [
                {
                  "type" : "index_not_found_exception",
                  "reason" : "no such index [test1]",
                  "index_uuid" : "6ZsTD9fJSDOJ8WiRURGf2A",
                  "index" : "test1"
                }
              ]
            }
          }
        }
      ]
    }
  }
}]
[2021-11-07T14:00:27,693][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=field_caps/10_basic/Get object and nested field caps] after test
REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.16.0#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=field_caps/10_basic/Get object and nested field caps}" -Dtests.seed=8F52471DA528A074 -Dtests.bwc=true -Dtests.locale=ga -Dtests.timezone=Africa/Asmera -Druntime.java=17

[cbuescher]

New one today: https://gradle-enterprise.elastic.co/s/sm6dnqrgqcunq/tests/:qa:mixed-cluster:v7.6.1%23mixedClusterTest/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT/test%20%7Bp0=field_caps%2F10_basic%2FGet%20object%20and%20nested%20field%20caps%7D?top-execution=1

[jkakavas]

Two more today : https://gradle-enterprise.elastic.co/s/7si2pr4dshbmq and https://gradle-enterprise.elastic.co/s/yh2sbheezc264

[cbuescher]

The two ones mentioned above on Nov 10th are slightly different failures though in other test cases in the same base test.
From https://gradle-enterprise.elastic.co/s/7si2pr4dshbmq (this is 10_basic/Get simple field caps) :

java.lang.AssertionError: Failure at [field_caps/10_basic:127]: field [fields.number.double.indices] is null

[2021-11-10T10:29:35,264][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=field_caps/10_basic/Get simple field caps] before test
[2021-11-10T10:29:35,550][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  "stash" : {
    "body" : {
      "indices" : [
        "test1",
        "test2",
        "test3"
      ],
      "fields" : {
        "geo" : {
          "geo_point" : {
            "type" : "geo_point",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : true
          }
        },
        "date" : {
          "date" : {
            "type" : "date",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : true
          }
        },
        "number" : {
          "double" : {
            "type" : "double",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : true
          }
        },
        "text" : {
          "text" : {
            "type" : "text",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : false
          }
        },
        "keyword" : {
          "keyword" : {
            "type" : "keyword",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : true
          }
        }
      }
    }
  }
}]
[2021-11-10T10:29:35,715][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=field_caps/10_basic/Get simple field caps] after test
REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.6.0#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=field_caps/10_basic/Get simple field caps}" -Dtests.seed=4E50B3AB0C2537FE -Dtests.bwc=true -Dtests.locale=hi -Dtests.timezone=America/Nipigon -Druntime.java=8

And from https://gradle-enterprise.elastic.co/s/yh2sbheezc264 (this is 10_basic/Field caps with include_unmapped)

Failure at [field_caps/10_basic:317]: field [fields.text.text.indices] has a true value but it shouldn't

[2021-11-10T08:19:53,380][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=field_caps/10_basic/Field caps with include_unmapped] before test
[2021-11-10T08:19:53,789][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  "stash" : {
    "body" : {
      "indices" : [
        "test1",
        "test2",
        "test3"
      ],
      "fields" : {
        "text" : {
          "text" : {
            "type" : "text",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : false,
            "indices" : [
              "test1",
              "test2"
            ]
          },
          "unmapped" : {
            "type" : "unmapped",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false,
            "indices" : [
              "test3"
            ]
          }
        },
        "misc" : {
          "text" : {
            "type" : "text",
            "metadata_field" : false,
            "searchable" : true,
            "aggregatable" : false,
            "indices" : [
              "test1"
            ]
          },
          "unmapped" : {
            "type" : "unmapped",
            "metadata_field" : false,
            "searchable" : false,
            "aggregatable" : false,
            "indices" : [
              "test2",
              "test3"
            ]
          }
        }
      }
    }
  }
}]
[2021-11-10T08:19:54,508][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=field_caps/10_basic/Field caps with include_unmapped] after test
REPRODUCE WITH: ./gradlew ':qa:mixed-cluster:v7.10.1#mixedClusterTest' -Dtests.class="org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT" -Dtests.method="test {p0=field_caps/10_basic/Field caps with include_unmapped}" -Dtests.seed=242020028CC26D77 -Dtests.bwc=true -Dtests.locale=es-CU -Dtests.timezone=Europe/Budapest -Druntime.java=8

[cbuescher]

I've looked a bit more at the failure history of p0=field_caps/10_basic/Get object and nested field caps and saw first irregularities with "null" values or unexpected list of indices as early as Oct 16th with e.g. this run https://gradle-enterprise.elastic.co/s/n2vucmljn2262

Here we have an see a failure in the field_caps response for the missing index:

 "failures" : [
        {
          "indices" : [
            "test2"
          ],
          "failure" : {
            "error" : {
              "root_cause" : [
                {
                  "type" : "i_o_exception",
                  "reason" : "Unknown Option ordinal [116]"
                }
              ],
              "type" : "i_o_exception",
              "reason" : "Unknown Option ordinal [116]"
            }
          }
        }
      ]

Since the start of these series of occasaional failures lies shortly after the merge of #79212 I suspect some changes in error handling/propagation to come into play here. I'm still looking at reproducing the error locally but maybe @dnhatn has some theory what might be happening here and how we can sometimes get these incomplete results after merging the responses.

[cbuescher]

"reason" : "Unknown Option ordinal [116]"

This suggests something broke in the StreamInput#readEnum for IndicesOptions enum, but that might be a one-off, I haven't seen these kind of errors in the other test failures so far. So maybe a red hering, but the other cases remain suspect

[dnhatn]

@cbuescher Thanks for your investigation and the ping. I will look into the failure.