[SPARK-32160][CORE][PYSPARK] Add a config to switch allow/disallow to create SparkContext in executors. #29278

ueshin · 2020-07-29T01:51:53Z

What changes were proposed in this pull request?

This is a follow-up of #28986.
This PR adds a config to switch allow/disallow to create SparkContext in executors.

spark.driver.allowSparkContextInExecutors

Why are the changes needed?

Some users or libraries actually create SparkContext in executors.
We shouldn't break their workloads.

Does this PR introduce any user-facing change?

Yes, users will be able to create SparkContext in executors with the config enabled.

How was this patch tested?

More tests are added.

…ors.

ueshin · 2020-07-29T01:54:36Z

core/src/main/scala/org/apache/spark/internal/config/package.scala

+  private[spark] val ALLOW_SPARK_CONTEXT_IN_EXECUTORS =
+    ConfigBuilder("spark.driver.allowSparkContextInExecutors")
+      .doc("If set to true, SparkContext can be created in executors.")
+      .version("3.0.1")


I'll submit another PR with the opposite default value against branch-3.0 when this is good to go.

Also I'll add a migration guide later.

SparkQA · 2020-07-29T04:48:52Z

Test build #126746 has finished for PR 29278 at commit 756837d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-30T01:44:20Z

Test build #126790 has finished for PR 29278 at commit 77a086a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-30T02:27:00Z

Test build #126792 has finished for PR 29278 at commit 1c1b7ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2020-07-30T19:32:23Z

hmm. A test is still failing in https://github.com/apache/spark/pull/29278/checks?check_run_id=928797731:

[info] - SPARK-32175: Plugin initialization should start after heartbeater started *** FAILED *** (32 seconds, 952 milliseconds)
[info]   The code passed to failAfter did not complete within 30000000000 nanoseconds. (SparkSubmitSuite.scala:1438)
[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info]   at java.lang.Thread.getStackTrace(Thread.java:1559)
[info]   at org.scalatest.concurrent.TimeLimits.failAfterImpl(TimeLimits.scala:234)
[info]   at org.scalatest.concurrent.TimeLimits.failAfterImpl$(TimeLimits.scala:233)
[info]   at org.apache.spark.deploy.SparkSubmitSuite$.failAfterImpl(SparkSubmitSuite.scala:1419)
[info]   at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:230)
[info]   at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:229)
[info]   at org.apache.spark.deploy.SparkSubmitSuite$.failAfter(SparkSubmitSuite.scala:1419)
[info]   at org.apache.spark.deploy.SparkSubmitSuite$.runSparkSubmit(SparkSubmitSuite.scala:1438)
[info]   at org.apache.spark.executor.ExecutorSuite.$anonfun$new$17(ExecutorSuite.scala:469)
[info]   at org.apache.spark.executor.ExecutorSuite.$anonfun$new$17$adapted(ExecutorSuite.scala:407)
[info]   at org.apache.spark.SparkFunSuite.withTempDir(SparkFunSuite.scala:170)
[info]   at org.apache.spark.executor.ExecutorSuite.$anonfun$new$16(ExecutorSuite.scala:407)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)

...

[info] *** 1 TEST FAILED ***
[error] Failed: Total 2869, Failed 1, Errors 0, Passed 2868, Ignored 8, Canceled 1
[error] Failed tests:
[error] 	org.apache.spark.executor.ExecutorSuite
[error] (core/test:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1970 s, completed Jul 30, 2020 6:28:55 PM

but I can see the failure in other PRs, e.g., https://github.com/apache/spark/pull/29276/checks?check_run_id=924258317.
Also it passes in my local.

I'll rerun the tests.

SparkQA · 2020-07-31T00:57:35Z

Test build #126812 has finished for PR 29278 at commit ce5860f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-31T07:05:02Z

Test build #126839 has finished for PR 29278 at commit 0ea7ea9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

ueshin · 2020-07-31T07:12:41Z

Jenkins, retest this please.

HyukjinKwon · 2020-07-31T08:28:13Z

GitHub Actiosn build passed. Merged to master.

SparkQA · 2020-07-31T12:08:16Z

Test build #126869 has finished for PR 29278 at commit 0ea7ea9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…ow to create SparkContext in executors ### What changes were proposed in this pull request? This is a backport of #29278, but with allowing to create `SparkContext` in executors by default. This PR adds a config to switch allow/disallow to create `SparkContext` in executors. - `spark.driver.allowSparkContextInExecutors` ### Why are the changes needed? Some users or libraries actually create `SparkContext` in executors. We shouldn't break their workloads. ### Does this PR introduce _any_ user-facing change? Yes, users will be able to disallow to create `SparkContext` in executors with the config disabled. ### How was this patch tested? More tests are added. Closes #29294 from ueshin/issues/SPARK-32160/3.0/add_configs. Authored-by: Takuya UESHIN <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

dongjoon-hyun

Hi, @ueshin . Why do we use spark.driver namespace in this new configuration? Can we use spark.executor namespace instead because this is irrelevant to Spark Driver.

spark.driver.allowSparkContextInExecutors

Since this is a configuration namespace issue, cc @cloud-fan and @gatorsmile .

I guess we can change like this simply.

- spark.driver.allowSparkContextInExecutors
+ spark.executor.allowSparkContext

ueshin · 2020-08-03T20:08:37Z

@Dooyoung-Hwang The name was inspired by the old config spark.driver.allowMultipleContexts in the comment #28986 (comment).
But, yeah, spark.executor.allowSparkContext sounds more sense. cc @cloud-fan, @gatorsmile

ueshin · 2020-08-03T21:33:15Z

I submitted PRs to rename the config #29340, #29341.

dongjoon-hyun · 2020-08-03T21:34:41Z

Thanks, @ueshin . BTW, my name is Dongjoon Hyun with id @dongjoon-hyun . :)

ueshin · 2020-08-03T21:37:09Z

Oops, sorry @dongjoon-hyun and @Dooyoung-Hwang. I mistakenly pick up the id from the suggested id list. 🙏

HyukjinKwon · 2020-08-04T01:28:55Z

Actually, what @dongjoon-hyun pointed out makes sense to me. Why don't we just change?

HyukjinKwon · 2020-08-04T01:29:14Z

oh the PRs are already open.

… switch allow/disallow SparkContext in executors ### What changes were proposed in this pull request? This is a follow-up of #29294. This PR changes the config name to switch allow/disallow `SparkContext` in executors as per the comment #29278 (review). ### Why are the changes needed? The config name `spark.executor.allowSparkContext` is more reasonable. ### Does this PR introduce _any_ user-facing change? Yes, the config name is changed. ### How was this patch tested? Updated tests. Closes #29341 from ueshin/issues/SPARK-32160/3.0/change_config_name. Authored-by: Takuya UESHIN <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

…ch allow/disallow SparkContext in executors ### What changes were proposed in this pull request? This is a follow-up of #29278. This PR changes the config name to switch allow/disallow `SparkContext` in executors as per the comment #29278 (review). ### Why are the changes needed? The config name `spark.executor.allowSparkContext` is more reasonable. ### Does this PR introduce _any_ user-facing change? Yes, the config name is changed. ### How was this patch tested? Updated tests. Closes #29340 from ueshin/issues/SPARK-32160/change_config_name. Authored-by: Takuya UESHIN <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

Add configs to switch allow/disallow to create SparkContext in execut…

756837d

…ors.

ueshin requested review from gatorsmile and srowen July 29, 2020 01:51

probot-autolabeler bot added CORE PYTHON SQL labels Jul 29, 2020

ueshin changed the title ~~[SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors.~~ [WIP][SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors. Jul 29, 2020

ueshin commented Jul 29, 2020

View reviewed changes

ueshin mentioned this pull request Jul 29, 2020

[SPARK-32160][CORE][PYSPARK] Disallow to create SparkContext in executors. #28986

Closed

ueshin added 3 commits July 29, 2020 15:49

Fix.

77a086a

Merge branch 'master' into issues/SPARK-32160/add_configs

27c0073

Add a migration guide.

1c1b7ec

probot-autolabeler bot added the DOCS label Jul 29, 2020

ueshin changed the title ~~[WIP][SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors.~~ [SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors. Jul 29, 2020

ueshin mentioned this pull request Jul 29, 2020

[SPARK-32160][CORE][PYSPARK][3.0] Add a config to switch allow/disallow to create SparkContext in executors. #29294

Closed

Use a single config.

9feb7e1

ueshin changed the title ~~[SPARK-32160][CORE][PYSPARK] Add configs to switch allow/disallow to create SparkContext in executors.~~ [SPARK-32160][CORE][PYSPARK] Add a config to switch allow/disallow to create SparkContext in executors. Jul 30, 2020

HyukjinKwon approved these changes Jul 30, 2020

View reviewed changes

Rerun tests.

ce5860f

Rerun tests.

0ea7ea9

HyukjinKwon closed this in 8014b0b Jul 31, 2020

dongjoon-hyun reviewed Aug 3, 2020

View reviewed changes

This was referenced Aug 3, 2020

[SPARK-32160][CORE][PYSPARK][FOLLOWUP] Change the config name to switch allow/disallow SparkContext in executors. #29340

Closed

[SPARK-32160][CORE][PYSPARK][3.0][FOLLOWUP] Change the config name to switch allow/disallow SparkContext in executors. #29341

Closed

[SPARK-32160][CORE][PYSPARK] Add a config to switch allow/disallow to create SparkContext in executors. #29278

[SPARK-32160][CORE][PYSPARK] Add a config to switch allow/disallow to create SparkContext in executors. #29278

Uh oh!

Conversation

ueshin commented Jul 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

ueshin Jul 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ueshin Jul 29, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 29, 2020

Uh oh!

SparkQA commented Jul 30, 2020

Uh oh!

SparkQA commented Jul 30, 2020

Uh oh!

ueshin commented Jul 30, 2020

Uh oh!

SparkQA commented Jul 31, 2020

Uh oh!

SparkQA commented Jul 31, 2020

Uh oh!

ueshin commented Jul 31, 2020

Uh oh!

HyukjinKwon commented Jul 31, 2020

Uh oh!

SparkQA commented Jul 31, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

ueshin commented Aug 3, 2020

Uh oh!

ueshin commented Aug 3, 2020

Uh oh!

dongjoon-hyun commented Aug 3, 2020

Uh oh!

ueshin commented Aug 3, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Aug 4, 2020

Uh oh!

HyukjinKwon commented Aug 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ueshin commented Jul 29, 2020 •

edited

Loading

ueshin Jul 29, 2020 •

edited

Loading

ueshin commented Aug 3, 2020 •

edited

Loading