Make test more robust for API key auth 429 #59077
Conversation
ywangd
added
>test
|
Pinging @elastic/es-security (:Security/Authentication) |
|
@tvernum I came to realise that we can configure the queue size to be smaller, e.g. |
Adds error handling when filling up the queue of the crypto thread pool. Also reduce queue size of the crypto thread pool to 10 so that the queue can be cleared out in time. Test testAuthenticationReturns429WhenThreadPoolIsSaturated has seen failure on CI when it tries to push 1000 tasks into the queue (setup phase). Since multiple tests share the same internal test cluster, it may be possible that there are lingering requests not fully cleared out from the queue. When it happens, we will not be able to push all 1000 tasks into the queue. But since what we need is just queue saturation, so as long as we can be sure that the queue is fully filled, it is safe to ignore rejection error and just move on. A number of 1000 tasks also take some to clear out, which could cause the test suite to time out. This PR change the queue to 10 so the tests would have better chance to complete in time.
Adds error handling when filling up the queue of the crypto thread pool. Also reduce queue size of the crypto thread pool to 10 so that the queue can be cleared out in time. Test testAuthenticationReturns429WhenThreadPoolIsSaturated has seen failure on CI when it tries to push 1000 tasks into the queue (setup phase). Since multiple tests share the same internal test cluster, it may be possible that there are lingering requests not fully cleared out from the queue. When it happens, we will not be able to push all 1000 tasks into the queue. But since what we need is just queue saturation, so as long as we can be sure that the queue is fully filled, it is safe to ignore rejection error and just move on. A number of 1000 tasks also take some to clear out, which could cause the test suite to time out. This PR change the queue to 10 so the tests would have better chance to complete in time.
|
@ywangd these still seem to time out now, see https://gradle-enterprise.elastic.co/s/4gfed632tmtcc/tests/:x-pack:plugin:security:test/org.elasticsearch.xpack.security.authc.ApiKeyIntegTests/testAuthenticationReturns429WhenThreadPoolIsSaturated#1 |
|
Timouts similarly on master: https://gradle-enterprise.elastic.co/s/qnxbule5yyo6s/console-log?task=:x-pack:plugin:security:test#L3817 |
5 participants
This PR adds error catching when filling up the queue of the crypto thread pool.
Test
testAuthenticationReturns429WhenThreadPoolIsSaturatedhas seen failure on CI when it tries to push 1000 tasks into the queue (setup phase). Since multiple tests share the same internal test cluster, it may be possible that there are lingering requests not fully cleared out from the queue. When it happens, we will not be able to push all 1000 tasks into the queue. But since what we need is just queue saturation, so as long as we can be sure that the queue is fully filled, it is safe to ignore rejection error and just move on.A number of 1000 tasks also take some to clear out, which could cause the test suite to time out. This PR change the queue to 10 so the tests would have better chance to complete in time.
Build scan: