Add test retry logic for flaky tests

[mtrezza]

New Feature / Enhancement Checklist

• I am not disclosing a vulnerability.
• I am not just asking a question.
• I have searched through existing issues.

Current Limitation

There are still some flaky tests;

• requires more effort to merge a PR
• shows ci: failed labels on README

Feature / Enhancement Description

Add a test-retry logic on jasmine-level (not GitHub workflow level) specifically for flaky tests.

1. Implement logic to retry specific tests if they fail, in the style of the existing custom test specifiers:
   

   parse-server/spec/helper.js

   Line 422 in 1850be4

   global.it_exclude_dbs = excluded => {

2. Identify flaky tests
3. Use retry-test-specifier for flaky tests

Alternatives / Workarounds

Manually re-run tests (status quo)

References

• Fix flaky Postgres tests #8613
• Fix flaky tests #7180
• Review skipped tests #8184
• https://gist.github.com/joeyparrish/c8b020a33095ea7cf191c06f4339a4dc

[parse-github-assistant]

Thanks for opening this issue!

• 🎉 We are excited about your ideas for improvement!

[dplewis]

@mtrezza I was just looking into doing something like this. Using the test name as an identifier would be the easiest way to go.

[mtrezza]

There may be ways; I've looked into this quite extensively with jasmine but I couldn't find a way given its architecture. Hence I published #8655 to document my findings, closed it and kept this issue open, in case someone finds a way in the future. I think I've spent more than a day on it. The challenge is to build something on top of, or integrated into jasmine, so we have the test stats, etc. It may not be difficult to re-run a test, but I couldn't find a way to do that so that jasmine understands what's happening.

Another approach that came to my mind was to maybe use the GitHub CI to automatically create a test run per spec file. So that instead of re-running all tests when one fails, only the spec file of the failing test needs to re-run. But this'd need to consider coverage reporting, not spamming our CI dashboard, etc.

[parseplatformorg]

🎉 This change has been released in version 7.3.0-alpha.7

[parseplatformorg]

🎉 This change has been released in version 7.3.0-beta.1

[parseplatformorg]

🎉 This change has been released in version 7.3.0