Release tests failing with unknown setting [xpack.searchable.snapshot.cache.size]

[davidkyle]

Build scan:
https://gradle-enterprise.elastic.co/s/mkclmahghcbgq
https://gradle-enterprise.elastic.co/s/b6ingm7sbiafa

Repro line:

 ./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsUuidValidationIntegTests.testMountFailsIfSnapshotChanged" \
  -Dtests.seed=96AC46B5E49A1EA5 \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=ar-TN \
  -Dtests.timezone=Asia/Novosibirsk \
  -Druntime.java=11

Reproduces locally?:

Applicable branches:
Master, 7.x

Failure history:
Regularly since 2nd September

First occurence - https://gradle-enterprise.elastic.co/s/5tyix4dniwqxm

Failure excerpt:

java.lang.IllegalArgumentException: unknown setting [xpack.searchable.snapshot.cache.size] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544)
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489)
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460)
	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431)
	at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149)
	at org.elasticsearch.node.Node.<init>(Node.java:392)
	at org.elasticsearch.node.MockNode.<init>(MockNode.java:99)
	at org.elasticsearch.node.MockNode.<init>(MockNode.java:89)
	at org.elasticsearch.test.InternalTestCluster.buildNode(InternalTestCluster.java:735)
	at org.elasticsearch.test.InternalTestCluster.getOrBuildRandomNode(InternalTestCluster.java:584)
	at org.elasticsearch.test.InternalTestCluster.client(InternalTestCluster.java:784)
	at org.elasticsearch.test.ESIntegTestCase.client(ESIntegTestCase.java:653)
	at org.elasticsearch.test.ESIntegTestCase.client(ESIntegTestCase.java:646)
	at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:563)
	at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2233)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

[tlrx]

I opened #61888 last week, thinking that it would solve this issue.

@mark-vieira do you have an idea why #61888 was not sufficient to reenable the snapshot feature flag in release tests?

[mark-vieira]

Ah, sorry, I probably should ahve caught that in the review. The linked PR only set the flag for rest tests. What is failing above is internalClusterTest so you'll need to apply the system property to that task since those are embedded cluster tests.

[tlrx]

Thanks @mark-vieira.

While looking at this again today I noticed it changed again yesterday with #61879 but the releases tests are still failing.

I'm going to assign this to you and @jakelandis as it seems to be more relevant to your area.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)