[CI] AzureStorageCleanupThirdPartyTests failing, possibly needs new set of credentials

[cbuescher]

CI Link

https://gradle-enterprise.elastic.co/s/sevrq5jszpe3w

Repro line

./gradlew ':plugins:repository-azure:azureThirdPartyTest' --tests "org.elasticsearch.repositories.azure.AzureStorageCleanupThirdPartyTests.testListChildren" \ -Dtests.seed=203404662522F345 \ -Dtests.locale=de-CH \ -Dtests.timezone=Europe/Podgorica \ -Druntime.java=8

Does it reproduce?

Didn't try

Applicable branches

7.17, 8.0, 8.1

Failure history

Several failures today

Failure excerpt

org.elasticsearch.repositories.azure.AzureStorageCleanupThirdPartyTests > testListChildren FAILED
RepositoryVerificationException[[test-repo] path [7.17_third_party_tests_203404662522F345] is not accessible on master node]; nested: BlobStorageException[Status code 403, "AuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission.
RequestId:3ebf5395-201e-0023-1ff7-27a758000000
Time:2022-02-22T14:22:00.3240871Z"];
at __randomizedtesting.SeedInfo.seed([203404662522F345:68BCA86A44A7FC4D]:0)

    Caused by:
    com.azure.storage.blob.models.BlobStorageException: Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
    RequestId:3ebf5395-201e-0023-1ff7-27a758000000
    Time:2022-02-22T14:22:00.3240871Z</Message></Error>"

[cbuescher]

This looks very similar to #75648, I'll see if I can open a similar issue for Infra to solve this.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[cbuescher]

I opened https://github.com/elastic/infra/issues/34818 on the Infra side to see if they can check if this is indeed related to problems with old credentials.

[cbuescher]

These are probably also related, but the error looks slightly different: https://gradle-enterprise.elastic.co/s/euqoyry4qsqlq/console-log?task=:plugins:repository-azure:azureThirdPartyTest

org.elasticsearch.repositories.azure.AzureStorageCleanupThirdPartyTests > testCleanup FAILED
    RepositoryVerificationException[[test-repo] path [7.17_third_party_tests_3EE8C9A2B44BBD81] is not accessible on master node]; nested: BlobStorageException[Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>ResourceNotFound</Code><Message>The specified resource does not exist.
    RequestId:96a48058-501e-004b-4bf7-27c1c8000000
    Time:2022-02-22T14:20:46.0735493Z</Message></Error>"];
        at __randomizedtesting.SeedInfo.seed([3EE8C9A2B44BBD81:6EC29A0CCDE62D34]:0)

        Caused by:
        com.azure.storage.blob.models.BlobStorageException: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>ResourceNotFound</Code><Message>The specified resource does not exist.
        RequestId:96a48058-501e-004b-4bf7-27c1c8000000
        Time:2022-02-22T14:20:46.0735493Z</Message></Error>"

The error code in the nested exception is slightly different and it doesn't mention 'AuthorizationPermissionMismatch'. The other build before is a "SAS" build, whatever that might mean to anybody.

[mark-vieira]

See @drewr's comment here.

@elastic/es-distributed I realize this is "delivery-esque" since it's infra but if any ES dev can update these credentials my preference is for the owning area team to be self-sufficient here.

[mark-vieira]

@fcofdez are you still having trouble updating these credentials? Do we need to reach back out to infra?