[SPARK-7361][STREAMING] Throw unambiguous exception when attempting to start multiple StreamingContexts in the same JVM #5907

tdas · 2015-05-05T09:06:08Z

Currently attempt to start a streamingContext while another one is started throws a confusing exception that the action name JobScheduler is already registered. Instead its best to throw a proper exception as it is not supported.

AmplabJenkins · 2015-05-05T09:07:10Z

Merged build triggered.

AmplabJenkins · 2015-05-05T09:07:19Z

Merged build started.

tdas · 2015-05-05T09:12:59Z

@JoshRosen Since you have done this before the SparkContext, could you quickly take a look at this?

AmplabJenkins · 2015-05-05T09:22:20Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-05T09:22:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31854/
Test FAILed.

SparkQA · 2015-05-05T09:24:08Z

Test build #765 has started for PR 5907 at commit 5870e2b.

SparkQA · 2015-05-05T11:04:09Z

Test build #765 has finished for PR 5907 at commit 5870e2b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Not(child: Expression) extends UnaryExpression with Predicate with ExpectsInputTypes
- case class And(left: Expression, right: Expression)
- case class Or(left: Expression, right: Expression)
- abstract class BinaryComparison extends BinaryExpression with Predicate
- trait StringRegexExpression extends ExpectsInputTypes
- trait CaseConversionExpression extends ExpectsInputTypes
- case class Substring(str: Expression, pos: Expression, len: Expression)

JoshRosen · 2015-05-05T16:58:56Z

streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala

To clarify, we're allowed to start multiple StreamingContexts provided that only one context is running at a time, right? Since that's the case, what do you think about rewording this to something like "Only one StreamingContext may be running in this JVM"? I just worry that "started" might be misinterpreted as saying that you can only create one SparkContext per JVM.

We don't need to add an explicit test for starting a new StreamingContext after stopping the old one since this case is implicitly handled by all of our other test suites.

Also, I think that core has some logic to print the call site of the active context in the error message, which can sometimes be a useful debugging aid when determining where contexts are being leaked.

Well, let me be a little bit more unambiguous in the choice of verbs. We are allowed to instantiate multiple StreamingContexts provided only one is started (ssc.start()) at a time. That's why I used "started" so that it directly maps to ssc.start(). This is different from SparkContexts where it is started as soon as it is instantiated.

I will add the callsite information.

AmplabJenkins · 2015-05-06T15:57:11Z

Build triggered.

AmplabJenkins · 2015-05-06T15:57:18Z

Build started.

SparkQA · 2015-05-06T15:57:54Z

Test build #31985 has started for PR 5907 at commit 5870e2b.

SparkQA · 2015-05-06T18:04:25Z

Test build #31985 has finished for PR 5907 at commit 5870e2b.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T18:04:30Z

Build finished. Test FAILed.

AmplabJenkins · 2015-05-06T18:04:31Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31985/
Test FAILed.

JoshRosen · 2015-05-06T18:07:58Z

This build break is my fault; a patch that I merged depended on a file that was removed while reverting another patch. Pushing a hotfix now to re-add that file.

JoshRosen · 2015-05-06T18:08:18Z

Jenkins, retest this please.

AmplabJenkins · 2015-05-06T18:12:11Z

Build triggered.

AmplabJenkins · 2015-05-06T18:12:20Z

Build started.

SparkQA · 2015-05-06T18:12:26Z

Test build #32007 has started for PR 5907 at commit 5870e2b.

SparkQA · 2015-05-06T20:08:45Z

Test build #32007 has finished for PR 5907 at commit 5870e2b.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T20:08:50Z

Build finished. Test FAILed.

AmplabJenkins · 2015-05-06T20:08:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32007/
Test FAILed.

Conflicts: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala

AmplabJenkins · 2015-05-07T08:57:14Z

Merged build triggered.

AmplabJenkins · 2015-05-07T08:57:22Z

Merged build started.

SparkQA · 2015-05-07T08:59:09Z

Test build #32097 has started for PR 5907 at commit 5fdfc0d.

AmplabJenkins · 2015-05-07T09:02:10Z

Merged build triggered.

AmplabJenkins · 2015-05-07T09:02:17Z

Merged build started.

SparkQA · 2015-05-07T09:02:31Z

Test build #32098 has started for PR 5907 at commit a9cd5bb.

SparkQA · 2015-05-07T09:51:47Z

Test build #32097 has finished for PR 5907 at commit 5fdfc0d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T09:51:51Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T09:51:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32097/
Test FAILed.

SparkQA · 2015-05-07T09:56:37Z

Test build #32098 has finished for PR 5907 at commit a9cd5bb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-07T09:56:41Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T09:56:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32098/
Test FAILed.

AmplabJenkins · 2015-05-07T18:17:12Z

Merged build triggered.

AmplabJenkins · 2015-05-07T18:17:22Z

Merged build started.

SparkQA · 2015-05-07T18:17:46Z

Test build #32130 has started for PR 5907 at commit fb81c4a.

AmplabJenkins · 2015-05-07T20:13:07Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-07T20:13:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32130/
Test FAILed.

tdas · 2015-05-07T20:13:36Z

test this again

SparkQA · 2015-05-07T22:54:23Z

Test build #781 has started for PR 5907 at commit fb81c4a.

SparkQA · 2015-05-08T00:45:13Z

Test build #781 has finished for PR 5907 at commit fb81c4a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-05-08T01:58:04Z

@JoshRosen any more thoughts?

JoshRosen · 2015-05-11T17:57:59Z

LGTM; thanks for adding the callsite.

tdas · 2015-05-11T17:58:33Z

Thanks @JoshRosen I am merging this.

…to start multiple StreamingContexts in the same JVM Currently attempt to start a streamingContext while another one is started throws a confusing exception that the action name JobScheduler is already registered. Instead its best to throw a proper exception as it is not supported. Author: Tathagata Das <[email protected]> Closes #5907 from tdas/SPARK-7361 and squashes the following commits: fb81c4a [Tathagata Das] Fix typo a9cd5bb [Tathagata Das] Added startSite to StreamingContext 5fdfc0d [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-7361 5870e2b [Tathagata Das] Added check for multiple streaming contexts (cherry picked from commit 1b46556) Signed-off-by: Tathagata Das <[email protected]>

…to start multiple StreamingContexts in the same JVM Currently attempt to start a streamingContext while another one is started throws a confusing exception that the action name JobScheduler is already registered. Instead its best to throw a proper exception as it is not supported. Author: Tathagata Das <[email protected]> Closes apache#5907 from tdas/SPARK-7361 and squashes the following commits: fb81c4a [Tathagata Das] Fix typo a9cd5bb [Tathagata Das] Added startSite to StreamingContext 5fdfc0d [Tathagata Das] Merge remote-tracking branch 'apache-github/master' into SPARK-7361 5870e2b [Tathagata Das] Added check for multiple streaming contexts

Added check for multiple streaming contexts

5870e2b

tdas changed the title ~~[SPARK-7361][STREAMING] Added check for multiple streaming contexts~~ [SPARK-7361][STREAMING] Throw unambiguous exception when attempting to start multiple StreamingContexts in the same JVM May 5, 2015

JoshRosen reviewed May 5, 2015
View reviewed changes

Merge remote-tracking branch 'apache-github/master' into SPARK-7361

5fdfc0d

Conflicts: streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala

Added startSite to StreamingContext

a9cd5bb

Fix typo

fb81c4a

asfgit closed this in 1b46556 May 11, 2015

[SPARK-7361][STREAMING] Throw unambiguous exception when attempting to start multiple StreamingContexts in the same JVM #5907

[SPARK-7361][STREAMING] Throw unambiguous exception when attempting to start multiple StreamingContexts in the same JVM #5907

Uh oh!

Conversation

tdas commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

tdas commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

AmplabJenkins commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

SparkQA commented May 5, 2015

Uh oh!

JoshRosen May 5, 2015

Choose a reason for hiding this comment

Uh oh!

JoshRosen May 5, 2015

Choose a reason for hiding this comment

Uh oh!

tdas May 7, 2015

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

JoshRosen commented May 6, 2015

Uh oh!

JoshRosen commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

SparkQA commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 6, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

SparkQA commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

SparkQA commented May 7, 2015

Uh oh!

SparkQA commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

SparkQA commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015

Uh oh!

AmplabJenkins commented May 7, 2015