[SPARK-7356][STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch. #5918

harishreedharan · 2015-05-05T20:32:18Z

This is meant to make the FlumePollingStreamSuite deterministic. Now we basically count the number of batches that have been completed - and then verify the results rather than sleeping for random periods of time.

…sing SparkSink's batch CountDownLatch.

AmplabJenkins · 2015-05-05T20:37:09Z

Merged build triggered.

AmplabJenkins · 2015-05-05T20:37:19Z

Merged build started.

SparkQA · 2015-05-05T20:39:01Z

Test build #31899 has started for PR 5918 at commit 1108804.

SparkQA · 2015-05-05T22:20:52Z

Test build #31899 has finished for PR 5918 at commit 1108804.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-05T22:20:57Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-05T22:20:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31899/
Test PASSed.

harishreedharan · 2015-05-05T22:38:01Z

Jenkins, test this please

AmplabJenkins · 2015-05-05T22:42:10Z

Merged build triggered.

AmplabJenkins · 2015-05-05T22:42:20Z

Merged build started.

SparkQA · 2015-05-05T22:43:02Z

Test build #31917 has started for PR 5918 at commit 1108804.

SparkQA · 2015-05-05T23:51:06Z

Test build #31917 has finished for PR 5918 at commit 1108804.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-05T23:51:11Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-05T23:51:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31917/
Test FAILed.

harishreedharan · 2015-05-06T00:05:15Z

This test passed, unresolved deps in some other test caused the failure.

[info] FlumePollingStreamSuite:
[info] - flume polling test (3 seconds, 42 milliseconds)
[info] - flume polling test multiple hosts (3 seconds, 3 milliseconds)

harishreedharan · 2015-05-06T00:05:23Z

Jenkins, test this please

AmplabJenkins · 2015-05-06T00:07:11Z

Merged build triggered.

AmplabJenkins · 2015-05-06T00:07:20Z

Merged build started.

SparkQA · 2015-05-06T00:09:02Z

Test build #31925 has started for PR 5918 at commit 1108804.

SparkQA · 2015-05-06T01:55:08Z

Test build #31925 has finished for PR 5918 at commit 1108804.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T01:55:14Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-06T01:55:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31925/
Test PASSed.

harishreedharan · 2015-05-06T05:17:43Z

Ping @tdas

harishreedharan · 2015-05-06T18:27:59Z

test this please

AmplabJenkins · 2015-05-06T18:32:10Z

Merged build triggered.

AmplabJenkins · 2015-05-06T18:32:20Z

Merged build started.

SparkQA · 2015-05-06T18:34:02Z

Test build #32011 has started for PR 5918 at commit 1108804.

SparkQA · 2015-05-06T19:34:59Z

Test build #32011 has finished for PR 5918 at commit 1108804.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T19:35:03Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-06T19:35:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32011/
Test FAILed.

harishreedharan · 2015-05-11T20:40:30Z

Sorry, clicked the wrong button :/

AmplabJenkins · 2015-05-11T20:42:12Z

Merged build triggered.

AmplabJenkins · 2015-05-11T20:42:20Z

Merged build started.

SparkQA · 2015-05-11T20:44:01Z

Test build #32419 has started for PR 5918 at commit 93f24f3.

SparkQA · 2015-05-11T21:52:33Z

Test build #32419 has finished for PR 5918 at commit 93f24f3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-11T21:52:38Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-11T21:52:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32419/
Test FAILed.

harishreedharan · 2015-05-11T21:54:39Z

Unrelated failure in org.apache.spark.streaming.BasicOperationsSuite.rdd cleanup - input blocks and persisted RDDs

harishreedharan · 2015-05-11T21:56:21Z

Jenkins, test this please

AmplabJenkins · 2015-05-11T21:57:11Z

Merged build triggered.

AmplabJenkins · 2015-05-11T21:57:20Z

Merged build started.

SparkQA · 2015-05-11T21:59:10Z

Test build #32427 has started for PR 5918 at commit 93f24f3.

SparkQA · 2015-05-11T23:40:45Z

Test build #32427 has finished for PR 5918 at commit 93f24f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-11T23:40:50Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-11T23:40:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32427/
Test PASSed.

harishreedharan · 2015-05-12T01:18:43Z

Jenkins, test this please

AmplabJenkins · 2015-05-12T01:22:12Z

Merged build triggered.

AmplabJenkins · 2015-05-12T01:22:22Z

Merged build started.

SparkQA · 2015-05-12T01:23:12Z

Test build #32446 has started for PR 5918 at commit 93f24f3.

SparkQA · 2015-05-12T03:15:01Z

Test build #32446 has finished for PR 5918 at commit 93f24f3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-12T03:15:09Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-12T03:15:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32446/
Test PASSed.

andrewor14 · 2015-05-13T23:43:16Z

I took a quick look and it looks reasonable. I'm going to merge this into master and 1.4 first and we'll see if it fails again. Thanks @harishreedharan.

…using SparkSink's batch CountDownLatch. This is meant to make the FlumePollingStreamSuite deterministic. Now we basically count the number of batches that have been completed - and then verify the results rather than sleeping for random periods of time. Author: Hari Shreedharan <[email protected]> Closes #5918 from harishreedharan/flume-test-fix and squashes the following commits: 93f24f3 [Hari Shreedharan] Add an eventually block to ensure that all received data is processed. Refactor the dstream creation and remove redundant code. 1108804 [Hari Shreedharan] [SPARK-7356][STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch. (cherry picked from commit 61d1e87) Signed-off-by: Andrew Or <[email protected]>

…using SparkSink's batch CountDownLatch. This is meant to make the FlumePollingStreamSuite deterministic. Now we basically count the number of batches that have been completed - and then verify the results rather than sleeping for random periods of time. Author: Hari Shreedharan <[email protected]> Closes apache#5918 from harishreedharan/flume-test-fix and squashes the following commits: 93f24f3 [Hari Shreedharan] Add an eventually block to ensure that all received data is processed. Refactor the dstream creation and remove redundant code. 1108804 [Hari Shreedharan] [SPARK-7356][STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch.

[SPARK-7356][STREAMING] Fix flakey tests in FlumePollingStreamSuite u…

1108804

…sing SparkSink's batch CountDownLatch.

harishreedharan closed this May 11, 2015

harishreedharan reopened this May 11, 2015

asfgit closed this in 61d1e87 May 13, 2015

[SPARK-7356][STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch. #5918

[SPARK-7356][STREAMING] Fix flakey tests in FlumePollingStreamSuite using SparkSink's batch CountDownLatch. #5918

Uh oh!

Conversation

harishreedharan commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

harishreedharan commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

harishreedharan commented May 6, 2015

Uh oh!

harishreedharan commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

harishreedharan commented May 6, 2015

Uh oh!

harishreedharan commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

harishreedharan commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

SparkQA commented May 11, 2015

Uh oh!

SparkQA commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

AmplabJenkins commented May 11, 2015

Uh oh!

harishreedharan commented May 11, 2015

Uh oh!