This repository was archived by the owner on Feb 25, 2025. It is now read-only.
Shuffle test order and repeat test runs once. #12275
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The tests we write must be resilient to the order in which they are run in the harness. That is, they must not rely on global state set by other tests that have already run in the process. Also, these tests must themselves be repeatable. That is, they must correctly clean up after themselves and be able to run successfully again in the same process. This patch adds some safeguards against (but does NOT guarantee) the addition of tests that violate the dictum. Additionally, test failures must be easily reproducible for folks investigating the test failure. Also, tests that assert correctness of unrelated code must not stop progress on the authors patch. This changes does not hinder reproducibility of test failures because the random seed is printed in the logs before running each test. Developers attempting to reproduce the failure locally can do the same via the following invocation `--gtest_shuffle --gtest_repeat=<the count> --gtest_random_seed=<seed from failing run>`. This change does introduce potential burden on patch authors that may see failures in unrelated code as a newly failing shuffle seed is used on their runs. To ameliorate this, we will formulate guidance for them to aggressively mark such tests as disabled and file bugs to enable the same. The test seed is intentionally kept low because it’s purpose is to test that individual tests are repeatable. It must not be used as a replacement for fuzzing.
chinmaygarde
force-pushed
the
random_shuffle
branch
from
300a632 to
d5f9ab9
Compare
Contributor
|
(The "fail" thin gwas to hold the tree closed while mklim and aam were sorting out Dart breakages - it's safe to ignore at this point, has been reverted) |
engine-flutter-autoroll
added a commit
to engine-flutter-autoroll/flutter
that referenced
this pull request
Sep 18, 2019
engine-flutter-autoroll
added a commit
to flutter/flutter
that referenced
this pull request
Sep 18, 2019
[email protected]:flutter/engine.git/compare/d1692d4cc703...6a96417 git log d1692d4..6a96417 --no-merges --oneline 2019-09-17 [email protected] Roll fuchsia/sdk/core/linux-amd64 from RRgw-... to F-g18... (flutter/engine#12326) 2019-09-17 [email protected] Account for root surface transformation on the surfaces managed by the external view embedder. (flutter/engine#11384) 2019-09-17 [email protected] Introduce FlutterFragmentActivity (flutter/engine#12305) 2019-09-17 [email protected] Shuffle test order and repeat test runs once. (flutter/engine#12275) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected] on the revert to ensure that a human is aware of the problem. To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
Inconnu08
pushed a commit
to Inconnu08/flutter
that referenced
this pull request
Sep 30, 2019
[email protected]:flutter/engine.git/compare/d1692d4cc703...6a96417 git log d1692d4..6a96417 --no-merges --oneline 2019-09-17 [email protected] Roll fuchsia/sdk/core/linux-amd64 from RRgw-... to F-g18... (flutter/engine#12326) 2019-09-17 [email protected] Account for root surface transformation on the surfaces managed by the external view embedder. (flutter/engine#11384) 2019-09-17 [email protected] Introduce FlutterFragmentActivity (flutter/engine#12305) 2019-09-17 [email protected] Shuffle test order and repeat test runs once. (flutter/engine#12275) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll Please CC [email protected] on the revert to ensure that a human is aware of the problem. To report a problem with the AutoRoller itself, please file a bug: https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
10 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The tests we write must be resilient to the order in which they are run in the
harness. That is, they must not rely on global state set by other tests that
have already run in the process. Also, these tests must themselves be
repeatable. That is, they must correctly clean up after themselves and be able
to run successfully again in the same process.
This patch adds some safeguards against (but does NOT guarantee) the addition of
tests that violate the dictum.
Additionally, test failures must be easily reproducible for folks investigating
the test failure. Also, tests that assert correctness of unrelated code must not
stop progress on the authors patch.
This changes does not hinder reproducibility of test failures because the random
seed is printed in the logs before running each test. Developers attempting to
reproduce the failure locally can do the same via the following invocation
--gtest_shuffle --gtest_repeat=<the count> --gtest_random_seed=<seed from failing run>.This change does introduce potential burden on patch authors that may see
failures in unrelated code as a newly failing shuffle seed is used on their
runs. To ameliorate this, we will formulate guidance for them to aggressively
mark such tests as disabled and file bugs to enable the same.
The test seed is intentionally kept low because it’s purpose is to test that
individual tests are repeatable. It must not be used as a replacement for
fuzzing.