Add resilience in TokenAuthIntegTests (#64757)

The testExpiredTokensDeletedAfterExpiration method assumed that
invalidating a refresh token immediately after the delete task was run
would behave as if the token did not exist.

However, the Elasticsearch concurrency controls do not guarantee that
behaviour. It is possible for the request that searches for the token
document the corresponds to the refresh token to find an invalidated
but not yet deleted document which is reflected in the API response.

This change makes the test resilient to this behaviour by wrapping the
assertion in an assertBusy

Resolves: #56903

Author: tvernum

x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/TokenAuthIntegTests.java

@@ -194,11 +194,26 @@ public void testExpiredTokensDeletedAfterExpiration() throws Exception {
         assertThat(invalidateAccessTokenResponse.getPreviouslyInvalidatedTokens(), equalTo(0));
         assertThat(invalidateAccessTokenResponse.getErrors(), empty());
 
+        // Weird testing behaviour ahead...
+        // invalidating by access token (above) is a Get, but invalidating by refresh token (below) is a Search
+        // In a multi node cluster, in a small % of cases, the search might find a document that has been invalidated but not yet deleted
+        // from that node's shard.
+        // Our assertion, therefore, is that an attempt to invalidate the (already invalidated) refresh token must not actually invalidate
+        // anything (concurrency controls must prevent that), nor may return any errors,
+        // but it might _temporarily_ find an "already invalidated" token.
+        final InvalidateTokenRequest invalidateRefreshTokenRequest = InvalidateTokenRequest.refreshToken(refreshToken);
         InvalidateTokenResponse invalidateRefreshTokenResponse = restClient.security().invalidateToken(
-            InvalidateTokenRequest.refreshToken(refreshToken), SECURITY_REQUEST_OPTIONS);
+            invalidateRefreshTokenRequest, SECURITY_REQUEST_OPTIONS);
         assertThat(invalidateRefreshTokenResponse.getInvalidatedTokens(), equalTo(0));
-        assertThat(invalidateRefreshTokenResponse.getPreviouslyInvalidatedTokens(), equalTo(0));
         assertThat(invalidateRefreshTokenResponse.getErrors(), empty());
+
+        // 99% of the time, this will already be zero, but if not ensure it goes to zero within the allowed timeframe
+        if (invalidateRefreshTokenResponse.getPreviouslyInvalidatedTokens() > 0) {
+            assertBusy(() -> {
+                var newResponse = restClient.security().invalidateToken(invalidateRefreshTokenRequest, SECURITY_REQUEST_OPTIONS);
+                assertThat(newResponse.getPreviouslyInvalidatedTokens(), equalTo(0));
+            });
+        }
     }
 
     public void testInvalidateAllTokensForUser() throws Exception {