testExpiredTokensDeletedAfterExpiration fails with "Error invalidating refresh_token"

[DaveCTurner]

Build scan: https://gradle-enterprise.elastic.co/s/m2opl5u64rlmc/console-log?task=:x-pack:plugin:security:internalClusterTest

Repro line: ./gradlew ':x-pack:plugin:security:internalClusterTest' --tests "org.elasticsearch.xpack.security.authc.TokenAuthIntegTests.testExpiredTokensDeletedAfterExpiration" -Dtests.seed=CAFFBB05C6B0FD31 -Dtests.security.manager=true -Dtests.locale=en-ZA -Dtests.timezone=America/Recife -Druntime.java=11

Reproduces locally?: No

Applicable branches: master

Failure history: Failing occasionally after a flurry of failures back in December.

Failure excerpt:

org.elasticsearch.xpack.security.authc.TokenAuthIntegTests > testExpiredTokensDeletedAfterExpiration FAILED |  
-- | --
  | java.lang.AssertionError: |  
  | Expected: an empty collection |  
  | but: <[org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=exception, reason=Error invalidating refresh_token: ]]> |  
  | at __randomizedtesting.SeedInfo.seed([CAFFBB05C6B0FD31:CBDEAF1CFDF6B220]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.security.authc.TokenAuthIntegTests.testExpiredTokensDeletedAfterExpiration(TokenAuthIntegTests.java:208)

[benwtrent]

Another failure:

https://gradle-enterprise.elastic.co/s/yb4ibytk5yxm4

org.elasticsearch.xpack.security.authc.TokenAuthIntegTests > testExpiredTokensDeletedAfterExpiration FAILED |  
  | java.lang.AssertionError: |  
  | Expected: an empty collection |  
  | but: <[org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=exception, reason=Error invalidating refresh_token: ]]> |  
  | at __randomizedtesting.SeedInfo.seed([1F4A47D2E1066A4F:1E6B53CBDA40255E]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.security.authc.TokenAuthIntegTests.testExpiredTokensDeletedAfterExpiration(TokenAuthIntegTests.java:208)

Ran the test locally 100+ times and it never failed. Didn't reproduce locally with the reproduce line either.

[dakrone]

Another failure today: https://gradle-enterprise.elastic.co/s/mnks6ax5j4rqm

Didn't reproduce for me with

./gradlew ':x-pack:plugin:security:internalClusterTest' --tests "org.elasticsearch.xpack.security.authc.TokenAuthIntegTests.testExpiredTokensDeletedAfterExpiration" -Dtests.seed=E11483774700580D -Dtests.security.manager=true -Dtests.locale=fr-CA -Dtests.timezone=Europe/Ulyanovsk -Druntime.java=8

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[benwtrent]

New failure here: https://gradle-enterprise.elastic.co/s/rriswqxwr2oug/

org.elasticsearch.xpack.security.authc.TokenAuthIntegTests > testExpiredTokensDeletedAfterExpiration FAILED |  
-- | --
  | java.lang.AssertionError: |  
  | Expected: an empty collection |  
  | but: <[ElasticsearchException[Error invalidating refresh_token: ]; nested: DocumentMissingException[[_doc][token_MqPOrEvOrpBXSDuVmnPdGXTl0yxJGPeEUtZ4jjluujo]: document missing];]>

[DaveCTurner]

Another: https://gradle-enterprise.elastic.co/s/x6736rdotaxm6/console-log?task=:x-pack:plugin:security:internalClusterTest

org.elasticsearch.xpack.security.authc.TokenAuthIntegTests > testExpiredTokensDeletedAfterExpiration FAILED
    java.lang.AssertionError: 
    Expected: an empty collection
         but: <[ElasticsearchException[Error invalidating refresh_token: ]; nested: DocumentMissingException[[_doc][token__l1b_LFDFkIm-MHLh0zK9cRx-R7BGvNv50B1j6I79_A]: document missing];]>
        at __randomizedtesting.SeedInfo.seed([4D42695174955180:4C637D484FD31E91]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.security.authc.TokenAuthIntegTests.testExpiredTokensDeletedAfterExpiration(TokenAuthIntegTests.java:232)
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
REPRODUCE WITH: ./gradlew ':x-pack:plugin:security:internalClusterTest' --tests "org.elasticsearch.xpack.security.authc.TokenAuthIntegTests.testExpiredTokensDeletedAfterExpiration" -Dtests.seed=4D42695174955180 -Dtests.security.manager=true -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=mk-MK -Dtests.timezone=US/Arizona -Druntime.java=8

[csoulios]

Another failure today: https://gradle-enterprise.elastic.co/s/i2yx2esavowiw

java.lang.AssertionError: 
Expected: an empty collection
     but: <[org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=exception, reason=Error invalidating refresh_token: ]]>
	at __randomizedtesting.SeedInfo.seed([1FE6A9684B54518B:1EC7BD7170121E9A]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)

https://gradle-enterprise.elastic.co/s/i2yx2esavowiw/tests/:x-pack:plugin:security:internalClusterTest/org.elasticsearch.xpack.security.authc.TokenAuthIntegTests/testExpiredTokensDeletedAfterExpiration?expanded-stacktrace=WyIwIl0#1

[hendrikmuhs]

another one: https://gradle-enterprise.elastic.co/s/vcmhn6w6gt22i

[davidkyle]

They keep on coming https://gradle-enterprise.elastic.co/s/2mtuwctu6asuu