ci: Add workflow to detect flaky tests #7497
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
size-limit report 📦
|
mydea
force-pushed
the
fn/flaky-test-detector
branch
from
4c3902a to
84e59c3
Compare
mydea
force-pushed
the
fn/flaky-test-detector
branch
from
9d362f4 to
881fbb8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a new script & workflow that can be run to detect flaky tests.
It runs tests a given amount of times (10 to try it out), and logs errors if they happen, else log the average run time (note this is a bit ~~ as it just takes the duration to run the given test, separated by run count - so parallelization will skew this a bit. Still it should give a decent insight how long a given test takes).
Update:
This now runs automatically on PRs that change any browser integration tests. It will run all changed integration tests 100x, separate from the rest of the integration tests.
note that this is not merge blocking, but will report if it fails.
You can also choose to run this manually via workflow_dispatch, in which case it will run all tests 5x (for now).