ClassificationIT testSetUpgradeMode_ExistingTaskGetsUnassigned Failure

[davidkyle]

https://gradle-enterprise.elastic.co/s/75oonfpb3j3mw

org.elasticsearch.ElasticsearchException: all shards failedClose stacktrace
at __randomizedtesting.SeedInfo.seed([B5428DD8494405BC:568FC3C8087049D8]:0)
at org.elasticsearch.xpack.core.ml.utils.ExceptionsHelper.serverError(ExceptionsHelper.java:55)
at org.elasticsearch.xpack.ml.action.TransportGetDataFrameAnalyticsStatsAction.lambda$searchStats$8(TransportGetDataFrameAnalyticsStatsAction.java:216)

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.ClassificationIT.testSetUpgradeMode_ExistingTaskGetsUnassigned" -Dtests.seed=B5428DD8494405BC -Dtests.security.manager=true -Dtests.locale=es-AR -Dtests.timezone=America/Argentina/Ushuaia -Dcompiler.java=14
HomeTestsorg.elasticsearch.xpack.ml.integration.ClassificationIT » testSetUpgradeMode_ExistingTaskGetsUnassigned

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[przemekwitek]

I'm already working on it, reassigned to myself.

[przemekwitek]

In the server logs it seems that changing the value of upgrade mode happens "concurrently" with creating .ml-stats index:

1854 [2020-04-15T07:16:26,442][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Starting to set [upgrade_mode] to [true] from [false]
1855 [2020-04-15T07:16:26,457][INFO ][o.e.c.m.MetadataCreateIndexService] [integTest-2] applying create index request using v1 templates [{".ml-state":{"order":0,"version":8000099,"index_patterns":[".ml-s     tate*"],"settings":{"index":{"hidden":"true","auto_expand_replicas":"0-1"}},"mappings":{"_doc":{"_meta":{"version":"8000099"},"enabled":false}},"aliases":{}}}]
1856 [2020-04-15T07:16:26,468][INFO ][o.e.c.m.MetadataCreateIndexService] [integTest-2] [.ml-state-000001] creating index, cause [api], templates [.ml-state], shards [1]/[1], mappings [_doc]
1857 [2020-04-15T07:16:26,509][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Executing cluster state update
1858 [2020-04-15T07:16:26,541][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Cluster update response built: true
1859 [2020-04-15T07:16:26,542][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Enabling upgrade mode, must isolate datafeeds
1860 [2020-04-15T07:16:26,542][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Isolating datafeeds: []
1861 [2020-04-15T07:16:26,543][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Isolated the datafeeds
1862 [2020-04-15T07:16:26,544][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Un-assigning persistent tasks : [ data_frame_analytics-classification_set_upgrade_mode ]
1863 [2020-04-15T07:16:26,642][INFO ][o.e.c.m.MetadataCreateIndexService] [integTest-2] applying create index request using v1 templates [{".ml-stats":{"order":0,"version":8000099,"index_patterns":[".ml-s     tats-*"],"settings":{"index":{"hidden":"true","number_of_shards":"1","auto_expand_replicas":"0-1"}},"mappings":{"_doc":{"_meta":{"version":"8.0.0"},"dynamic":false,"properties":{"iteration":{"type":"     integer"},"hyperparameters":{"properties":{"class_assignment_objective":{"type":"keyword"},"downsample_factor":{"type":"double"},"eta":{"type":"double"},"eta_growth_rate_per_tree":{"type":"double"},"     feature_bag_fraction":{"type":"double"},"max_attempts_to_add_tree":{"type":"integer"},"max_optimization_rounds_per_hyperparameter":{"type":"integer"},"max_trees":{"type":"integer"},"num_folds":{"type     ":"integer"},"num_splits_per_feature":{"type":"integer"},"regularization_depth_penalty_multiplier":{"type":"double"},"regularization_leaf_weight_penalty_multiplier":{"type":"double"},"regularization_     soft_tree_depth_limit":{"type":"double"},"regularization_soft_tree_depth_tolerance":{"type":"double"},"regularization_tree_size_penalty_multiplier":{"type":"double"}}},"job_id":{"type":"keyword"},"pa     rameters":{"properties":{"compute_feature_influence":{"type":"boolean"},"feature_influence_threshold":{"type":"double"},"method":{"type":"keyword"},"n_neighbors":{"type":"integer"},"outlier_fraction"     :{"type":"double"},"standardization_enabled":{"type":"boolean"}}},"peak_usage_bytes":{"type":"long"},"model_id":{"type":"keyword"},"node_id":{"type":"keyword"},"inference_count":{"type":"long"},"fail     ure_count":{"type":"long"},"missing_all_fields_count":{"type":"long"},"skipped_docs_count":{"type":"long"},"timestamp":{"type":"date"},"timing_stats":{"properties":{"elapsed_time":{"type":"long"},"it     eration_time":{"type":"long"}}},"test_docs_count":{"type":"long"},"training_docs_count":{"type":"long"},"type":{"type":"keyword"},"validation_loss":{"properties":{"fold_values":{"properties":{"fold":     {"type":"integer"},"values":{"type":"double"}}},"loss_type":{"type":"keyword"}}}}}},"aliases":{}}}]
1864 [2020-04-15T07:16:26,651][INFO ][o.e.c.m.MetadataCreateIndexService] [integTest-2] [.ml-stats-000001] creating index, cause [api], templates [.ml-stats], shards [1]/[1], mappings [_doc]
1865 [2020-04-15T07:16:26,681][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Waited for tasks to be unassigned
1866 [2020-04-15T07:16:26,681][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Completed upgrade mode request
1867 [2020-04-15T07:16:26,701][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Starting to set [upgrade_mode] to [false] from [true]
1868 [2020-04-15T07:16:26,702][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Executing cluster state update
1869 [2020-04-15T07:16:26,730][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Cluster update response built: true
1870 [2020-04-15T07:16:26,730][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Disabling upgrade mode, must wait for tasks to not have AWAITING_UPGRADE assignment
1871 [2020-04-15T07:16:26,811][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Done waiting for tasks to be out of AWAITING_UPGRADE
1872 [2020-04-15T07:16:26,812][INFO ][o.e.x.m.a.TransportSetUpgradeModeAction] [integTest-2] Completed upgrade mode request

Other than that, there is no indication of upgrade mode-related failure in the logs. Still the test fails as it cannot gather task stats.
I have two hypotheses so far:

1. the failure is related to the fact that we are trying to search the index (.ml-stats-000001) that is still being created.
2. the failure is related to the fact that we are trying to gather stats for the task that is just being unassigned.

[mark-vieira]

This test has failed many times over the past 24 hours. As a point of process, if we see multiple failures in a single day we should automatically mute.

On that note, I've muted this test in master (ecc8599) and 7.x (ba0a39c)

[davidkyle]

Thanks for muting @mark-vieira, we were collecting extra debug information and were about to mute

[mark-vieira]

Thanks for muting @mark-vieira, we were collecting extra debug information and were about to mute

Is there some kind of annotation that will still execute a test, but not trigger a failure? If not, perhaps it's worth adding such a thing?

[przemekwitek]

Is there some kind of annotation that will still execute a test, but not trigger a failure? If not, perhaps it's worth adding such a thing?

Sounds like a good idea for all the "add more logging" cases. I'll try to find out if it's there.

[davidkyle]

Is there some kind of annotation that will still execute a test, but not trigger a failure? If not, perhaps it's worth adding such a thing?

It is a very interesting idea as leaving the tests unmuted with extra debug logging is a common workflow.

I took a look at JUnit TestRules which I think could be used to catch assertion failures and re-throw them as AssumptionViolatedExceptions. Assumptions do not fail the test suite so we would have our debugged test failure without the build failure.

But, without the build failure how would I know that my flaky test had failed again? I would have to check the logs of every build for the assumption exception. There is a conflict between suppressing the failure allowing the build to succeed and being notified of the failure I can't see a way to have both. Unless we track assumption failures which would just complicated things as certain tests assume there are running on certain platforms etc.

[mark-vieira]

But, without the build failure how would I know that my flaky test had failed again?

That's a good question. This is a good use case for retrying tests. In this scenario we still get the failures and we identify the test as being flakey if the passes on subsequent runs. There's first-class support for this in Gradle, and it's on my todo list to investigate how we might integrate it into the build.

Right now we aggressively mute builds to reduce noise but this does have the side effect that the data collected from failures is potentially useful.