[SPARK-13053] [TEST] Unignore tests in InternalAccumulatorSuite #10969
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
January 27, 2016 16:39
This commit actually removes one of the tests, which test stage retries when only a subset of the original partitions are submitted. This scenario is difficult to simulate in tests without introducing flakiness.
There's a chance that we don't actually wait until we finish calling the callback since currently we don't wait for it to happen. This commit adds a way to do that.
andrewor14
changed the title
Contributor
Author
|
Test build #50286 has finished for PR 10969 at commit
|
Contributor
Author
|
retest this please |
Events are posted asynchronously so we may not have called onStageCompleted or onJobEnd etc. before asserting things. We should wait until all events have been processed before proceeding.
|
Test build #50315 has finished for PR 10969 at commit
|
|
Test build #50317 has finished for PR 10969 at commit
|
|
Test build #50328 has finished for PR 10969 at commit
|
Contributor
Author
|
retest this please |
|
Test build #50392 has finished for PR 10969 at commit
|
Contributor
Author
|
Merging into master. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These were ignored because they are incorrectly written; they don't actually trigger stage retries, which is what the tests are testing. These tests are now rewritten to induce stage retries through fetch failures.
Note: there were 2 tests before and now there's only 1. What happened? It turns out that the case where we only resubmit a subset of of the original missing partitions is very difficult to simulate in tests without potentially introducing flakiness. This is because the
DAGSchedulerremoves all map outputs associated with a given executor when this happens, and we will need multiple executors to trigger this case, and sometimes the scheduler still removes map outputs from all executors.