MlDistributedFailureIT testClusterWithTwoMlNodes_RunsDatafeed_GivenOriginalNodeGoesDown failure #65710
Description
Build scan:
- https://gradle-enterprise.elastic.co/s/owwgiqfcfvmpm
- https://gradle-enterprise.elastic.co/s/szmkmg6lirwbs
- https://gradle-enterprise.elastic.co/s/el775yclzmwtu
Repro line:
./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT.testClusterWithTwoMlNodes_RunsDatafeed_GivenOriginalNodeGoesDown" -Dtests.seed=EDD33680F671DE6F -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=Europe/Paris -Druntime.java=8
Reproduces locally?:
No.
Applicable branches:
7.x.
It fails at master branch as well, but with a different error message. I am not sure whehther they are related.
The build scan for master branch failure is: https://gradle-enterprise.elastic.co/s/rcoohbqmucb4e
and error message is:
java.lang.AssertionError: |
-- | --
| Expected: a value equal to or greater than <80000L> |
| but: <69165L> was less than <80000L> |
at org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT.testClusterWithTwoMlNodes_RunsDatafeed_GivenOriginalNodeGoesDown(MlDistributedFailureIT.java:487)
--
Failure history:
The error is an IOException happened during teardown time. Normally, I would consider this type of error transient and let it pass. However this is a new test that just added and it started to fail at the same day. So I am creating this issue in case there are some underlying issue about it.
Failure excerpt:
java.io.IOException: Node didn't close within 10 seconds. |
-- | --
at __randomizedtesting.SeedInfo.seed([EDD33680F671DE6F:4A9E02922FD46A84]:0) |
-- | --
| | at org.elasticsearch.test.InternalTestCluster$NodeAndClient.close(InternalTestCluster.java:1091) |
| | at org.elasticsearch.test.InternalTestCluster.stopNodesAndClients(InternalTestCluster.java:1854) |
| | at org.elasticsearch.test.InternalTestCluster.stopNodesAndClient(InternalTestCluster.java:1844) |
| | at org.elasticsearch.test.InternalTestCluster.stopNode(InternalTestCluster.java:1728) |
| | at org.elasticsearch.xpack.ml.integration.MlDistributedFailureIT.testClusterWithTwoMlNodes_RunsDatafeed_GivenOriginalNodeGoesDown(MlDistributedFailureIT.java:497) |
| | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) |
| | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) |
| | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) |
| | at java.lang.reflect.Method.invoke(Method.java:498) |
| | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) |
| | at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) |
| | at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) |
| | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) |
| | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |
| | at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) |
| | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |
| | at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) |
| | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |
| | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |
| | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |
| | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) |
| | at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) |
| | at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) |
| | at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) |
| | at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) |
| | at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) |
| | at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) |
| | at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) |
| | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |
| | at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) |
| | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |
| | at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) |
| | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |
| | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |
| | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |
| | at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) |
| | at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) |
| | at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) |
| | at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) |
| | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |
| | at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) |
| | at java.lang.Thread.run(Thread.java:748)