[CI]MultiVersionRepositoryAccessIT fails due rollup cleanups

[hendrikmuhs]

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.6+default-distro+bwc/BWC_VERSION=6.4.3,nodes=centos-7&&immutable/5/

Reproduction:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.6+default-distro+bwc/BWC_VERSION=6.4.3,nodes=centos-7&&immutable/5/

Failure:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:39825], URI [/_rollup/job/_all], status line [HTTP/1.1 404 Not Found]
    {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_rollup","index_uuid":"_na_","index":"_rollup"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_rollup","index_uuid":"_na_","index":"_rollup"},"status":404}

The reason for the failure is the cleanup code for rollups - which are not even used for this test - using an API introduced in 7.0.0 but not available for 6.x.

This issue has been see already: #36000 - fixed and than reverted. I could not find the reason and do not know the details, hope @polyfractal can help out.

/CC @original-brownbear (FYI, worked last on MultiVersionRepositoryAccessIT)

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Rollup)

[polyfractal]

Huh, weird. The fix was reverted because it didn't quite fix the issue, but we later fixed it for real and re-deprecated the old URL in a way that should have worked with 6.x: #36451

In that PR, it looks like ESRestTestCase was put back to /_rollup regardless of version, and the individual test calls were configured to check for old cluster (e.g. here and here) where relevant.

ESRestTestCase is still configured this way, which seems odd that it would just now be throwing these failures since it's been that way for a year? Did something change with how we run BWC tests? I'm unfortunately not too familiar with how BWC test work, they have always been a bit of a black box for me.

[astefan]

Another failure here: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.6+default-distro+bwc/BWC_VERSION=6.5.0,nodes=centos-7&&immutable/20/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.6+default-distro+bwc/BWC_VERSION=6.4.2,nodes=centos-7&&immutable/20/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.6+default-distro+bwc/BWC_VERSION=6.5.1,nodes=centos-7&&immutable/20/console

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.6+default-distro+bwc/BWC_VERSION=6.5.4,nodes=centos-7&&immutable/20/console

https://gradle-enterprise.elastic.co/s/rfgoz3dqezwcq

REPRODUCE WITH: ./gradlew ':qa:repository-multi-version:v6.5.0#Step1OldClusterTest' --tests "org.elasticsearch.upgrades.MultiVersionRepositoryAccessIT.testUpgradeMovesRepoToNewMetaVersion" \
  -Dtests.seed=912A67A3FE47D352 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Africa/Kinshasa \
  -Dtests.distribution=default \
  -Dcompiler.java=13 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:repository-multi-version:v6.5.0#Step1OldClusterTest' --tests "org.elasticsearch.upgrades.MultiVersionRepositoryAccessIT.testCreateAndRestoreSnapshot" \
  -Dtests.seed=912A67A3FE47D352 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Africa/Kinshasa \
  -Dtests.distribution=default \
  -Dcompiler.java=13 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:repository-multi-version:v6.5.0#Step1OldClusterTest' --tests "org.elasticsearch.upgrades.MultiVersionRepositoryAccessIT.testReadOnlyRepo" \
  -Dtests.seed=912A67A3FE47D352 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Africa/Kinshasa \
  -Dtests.distribution=default \
  -Dcompiler.java=13 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:repository-multi-version:v6.5.0#Step1OldClusterTest' --tests "org.elasticsearch.upgrades.MultiVersionRepositoryAccessIT.testUpgradeMovesRepoToNewMetaVersion" \
  -Dtests.seed=912A67A3FE47D352 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Africa/Kinshasa \
  -Dtests.distribution=default \
  -Dcompiler.java=13 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:repository-multi-version:v6.5.0#Step1OldClusterTest' --tests "org.elasticsearch.upgrades.MultiVersionRepositoryAccessIT.testCreateAndRestoreSnapshot" \
  -Dtests.seed=912A67A3FE47D352 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Africa/Kinshasa \
  -Dtests.distribution=default \
  -Dcompiler.java=13 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:repository-multi-version:v6.5.0#Step1OldClusterTest' --tests "org.elasticsearch.upgrades.MultiVersionRepositoryAccessIT.testCreateAndRestoreSnapshot" \
  -Dtests.seed=912A67A3FE47D352 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Africa/Kinshasa \
  -Dtests.distribution=default \
  -Dcompiler.java=13 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:repository-multi-version:v6.5.0#Step1OldClusterTest' --tests "org.elasticsearch.upgrades.MultiVersionRepositoryAccessIT.testReadOnlyRepo" \
  -Dtests.seed=912A67A3FE47D352 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Africa/Kinshasa \
  -Dtests.distribution=default \
  -Dcompiler.java=13 \
  -Druntime.java=8

REPRODUCE WITH: ./gradlew ':qa:repository-multi-version:v6.5.0#Step1OldClusterTest' --tests "org.elasticsearch.upgrades.MultiVersionRepositoryAccessIT.testReadOnlyRepo" \
  -Dtests.seed=912A67A3FE47D352 \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=Africa/Kinshasa \
  -Dtests.distribution=default \
  -Dcompiler.java=13 \
  -Druntime.java=8

[polyfractal]

Oof. Will try to reproduce locally today and see what's up.

[salvatore-campagna]

As stated above this is failing because of a missing api in version 6.x. Probably pending tasks are running on nodes with version 6.x where the cleanup api is not available.

Checking recent failures we don't see any failure for the three tests involved:testUpgradeMovesRepoToNewMetaVersion, testReadOnlyRepo, testCreateAndRestoreSnapshot.

The AwaitsFix currently is for another issue, #94459 which involves testCreateAndRestoreSnapshot.

I suggest to close this. Eventually, when the AwaitsFix is removed and all tests run again we will see if the original three test fail.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)