test: retries for flaky tests

[drdaz]

New Pull Request Checklist

• I am not disclosing a vulnerability.
• I am creating this PR in reference to an issue.

Issue Description

We have some generalised flakiness in our tests. This update to xctoolchain should help us to spot and debug flaky tests.

[parse-github-assistant]

Thanks for opening this pull request!

• ❌ Please edit your post and use the provided template when creating a new pull request. This helps everyone to understand your post better and asks for essential information to quicker review the pull request.

[cbaker6]

@drdaz I recommend updating the CI to Xcode 13.2 as I think it may provide more details about the tests.

[drdaz]

The change also seems to have made no difference here. Hmm...

[drdaz]

After some poking around, I've come to the conclusion that the retry changes do work here. They just don't help with all of our flaky tests. There are some with occasional logical failures; those will probably work as documented in this test log with a test that just executes XCTFail - there are 10 sets of output showing the failure.

The ones that crash (and these are the bulk of our problem) don't get retried, as can be seen in the current iOS build.

It's not quite the result we could have hoped for, but I definitely think it's worth including.

[mtrezza]

What should we do with this PR? The only flakiness I observed recently was in the macos tests

[mtrezza]

@drdaz Would you like to resolve the conflicts so we can see whether it works? We've had several changes on the main branch and a flaky test left in macOS, so maybe the issue can be solved with this.

[dplewis]

@mtrezza I have also ran into this issue with timeout failure.