[CI] ProfileDomainIntegTests testConcurrentCreationOfNewProfiles failing

[ywangd]

Build scan:
https://gradle-enterprise.elastic.co/s/nmjznurkclizw/tests/:x-pack:plugin:security:internalClusterTest/org.elasticsearch.xpack.security.profile.ProfileDomainIntegTests/testConcurrentCreationOfNewProfiles

Reproduction line:
./gradlew ':x-pack:plugin:security:internalClusterTest' --tests "org.elasticsearch.xpack.security.profile.ProfileDomainIntegTests.testConcurrentCreationOfNewProfiles" -Dtests.seed=DF7A58E7374EF2C7 -Dtests.locale=id-ID -Dtests.timezone=Europe/Ljubljana -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.security.profile.ProfileDomainIntegTests&tests.test=testConcurrentCreationOfNewProfiles

Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=339, name=Thread-17, state=RUNNABLE, group=TGRP-ProfileDomainIntegTests]


  Caused by: java.lang.AssertionError: caught error when creating new profile: org.elasticsearch.action.UnavailableShardsException: at least one primary shard for the index [.security-profile-8] is unavailable

    at __randomizedtesting.SeedInfo.seed([DF7A58E7374EF2C7]:0)
    at org.junit.Assert.fail(Assert.java:88)
    at org.elasticsearch.xpack.security.profile.ProfileDomainIntegTests.lambda$testConcurrentCreationOfNewProfiles$0(ProfileDomainIntegTests.java:321)
    at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[dakrone]

This keeps failing for one of my PRs against master, example build scan: https://gradle-enterprise.elastic.co/s/fyr6zpx3hsljg

[ywangd]

I muted with 9d83874