[SPARK-20209][SS] Execute next trigger immediately if previous batch took longer than trigger interval #17525
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #75504 has started for PR 17525 at commit |
brkyvz
reviewed
Apr 4, 2017
| */ | ||
| def nextBatchTime(now: Long): Long = { | ||
| now / intervalMs * intervalMs + intervalMs | ||
| if (intervalMs == 0) now else now / intervalMs * intervalMs + intervalMs |
Contributor
There was a problem hiding this comment.
the doc seems wrong btw, mind fixing it? nextBatchTime(nextBatchTime(0)) = 100 or am I understanding it wrong?
Contributor
Author
There was a problem hiding this comment.
spoken offline, this isnt wrong.
brkyvz
reviewed
Apr 4, 2017
| } | ||
| } | ||
|
|
||
| def isStreamWaitingAt(time: Long): Boolean = synchronized { |
Contributor
There was a problem hiding this comment.
mind adding docs on when these should be used?
|
Test build #75514 has finished for PR 17525 at commit
|
|
Test build #75520 has finished for PR 17525 at commit
|
|
Test build #75521 has finished for PR 17525 at commit
|
tdas
commented
Apr 4, 2017
| testStream(mapped, OutputMode.Complete)( | ||
| StartStream(ProcessingTime(100), triggerClock = clock), | ||
| AssertStreamExecThreadToWaitForClock(), | ||
| StartStream(ProcessingTime(1000), triggerClock = clock), |
Contributor
Author
There was a problem hiding this comment.
This test needed fixing because this manual clock test was configured such that first batch takes > 100 ms even though the trigger interval was 100 ms. This caused additional batch to be automatically executed without waiting for the manual clock to be advance, thus breaking certain assumptions in the test.
|
Test build #75524 has finished for PR 17525 at commit
|
brkyvz
reviewed
Apr 5, 2017
| clockIncrementInTrigger = 1500 | ||
| manualClock.setTime(2000) | ||
| eventually { | ||
| assert(lastTriggerTime === 3500) |
Contributor
There was a problem hiding this comment.
it was hard to understand that the test is actually testing that this value is 3500 instead of 2000. Could you add a quick comment?
Contributor
|
left one minor comment, otherwise LGTM |
Contributor
|
Thanks for the change. It is easier to understand when things are being triggered now. LGTM. |
|
Test build #75527 has finished for PR 17525 at commit
|
cenyuhai
pushed a commit
to cenyuhai/spark
that referenced
this pull request
Oct 8, 2017
…took longer than trigger interval ## What changes were proposed in this pull request? For large trigger intervals (e.g. 10 minutes), if a batch takes 11 minutes, then it will wait for 9 mins before starting the next batch. This does not make sense. The processing time based trigger policy should be to do process batches as fast as possible, but no faster than 1 in every trigger interval. If batches are taking longer than trigger interval anyways, then no point waiting extra trigger interval. In this PR, I modified the ProcessingTimeExecutor to do so. Another minor change I did was to extract our StreamManualClock into a separate class so that it can be used outside subclasses of StreamTest. For example, ProcessingTimeExecutorSuite does not need to create any context for testing, just needs the StreamManualClock. ## How was this patch tested? Added new unit tests to comprehensively test this behavior. Author: Tathagata Das <[email protected]> Closes apache#17525 from tdas/SPARK-20209. (cherry picked from commit dad499f)
cenyuhai
pushed a commit
to cenyuhai/spark
that referenced
this pull request
Oct 8, 2017
…took longer than trigger interval ## What changes were proposed in this pull request? For large trigger intervals (e.g. 10 minutes), if a batch takes 11 minutes, then it will wait for 9 mins before starting the next batch. This does not make sense. The processing time based trigger policy should be to do process batches as fast as possible, but no faster than 1 in every trigger interval. If batches are taking longer than trigger interval anyways, then no point waiting extra trigger interval. In this PR, I modified the ProcessingTimeExecutor to do so. Another minor change I did was to extract our StreamManualClock into a separate class so that it can be used outside subclasses of StreamTest. For example, ProcessingTimeExecutorSuite does not need to create any context for testing, just needs the StreamManualClock. ## How was this patch tested? Added new unit tests to comprehensively test this behavior. Author: Tathagata Das <[email protected]> Closes apache#17525 from tdas/SPARK-20209. (cherry picked from commit dad499f)
cenyuhai
pushed a commit
to cenyuhai/spark
that referenced
this pull request
Oct 8, 2017
…took longer than trigger interval ## What changes were proposed in this pull request? For large trigger intervals (e.g. 10 minutes), if a batch takes 11 minutes, then it will wait for 9 mins before starting the next batch. This does not make sense. The processing time based trigger policy should be to do process batches as fast as possible, but no faster than 1 in every trigger interval. If batches are taking longer than trigger interval anyways, then no point waiting extra trigger interval. In this PR, I modified the ProcessingTimeExecutor to do so. Another minor change I did was to extract our StreamManualClock into a separate class so that it can be used outside subclasses of StreamTest. For example, ProcessingTimeExecutorSuite does not need to create any context for testing, just needs the StreamManualClock. ## How was this patch tested? Added new unit tests to comprehensively test this behavior. Author: Tathagata Das <[email protected]> Closes apache#17525 from tdas/SPARK-20209. (cherry picked from commit dad499f)
cenyuhai
pushed a commit
to cenyuhai/spark
that referenced
this pull request
Oct 8, 2017
…took longer than trigger interval ## What changes were proposed in this pull request? For large trigger intervals (e.g. 10 minutes), if a batch takes 11 minutes, then it will wait for 9 mins before starting the next batch. This does not make sense. The processing time based trigger policy should be to do process batches as fast as possible, but no faster than 1 in every trigger interval. If batches are taking longer than trigger interval anyways, then no point waiting extra trigger interval. In this PR, I modified the ProcessingTimeExecutor to do so. Another minor change I did was to extract our StreamManualClock into a separate class so that it can be used outside subclasses of StreamTest. For example, ProcessingTimeExecutorSuite does not need to create any context for testing, just needs the StreamManualClock. ## How was this patch tested? Added new unit tests to comprehensively test this behavior. Author: Tathagata Das <[email protected]> Closes apache#17525 from tdas/SPARK-20209. (cherry picked from commit dad499f)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
For large trigger intervals (e.g. 10 minutes), if a batch takes 11 minutes, then it will wait for 9 mins before starting the next batch. This does not make sense. The processing time based trigger policy should be to do process batches as fast as possible, but no faster than 1 in every trigger interval. If batches are taking longer than trigger interval anyways, then no point waiting extra trigger interval.
In this PR, I modified the ProcessingTimeExecutor to do so. Another minor change I did was to extract our StreamManualClock into a separate class so that it can be used outside subclasses of StreamTest. For example, ProcessingTimeExecutorSuite does not need to create any context for testing, just needs the StreamManualClock.
How was this patch tested?
Added new unit tests to comprehensively test this behavior.