[SPARK-27036][SPARK-SQL] Cancel the running jobs in the background if broadcast future timeout error occurs #24036

sujith71955 · 2019-03-09T07:48:05Z

What changes were proposed in this pull request?

Currently even Broadcast thread is timed out, Jobs are not aborted and it will run in the background.
As per current design the broadcast future will be waiting till the timeout for the job result, which needs to be broadcasted , when the broadcast future timeout happens the
job tasks running in the background will not getting killed and it will continue running in background.

As part of solution we shall get the jobs based on execution id from app-status store and cancel the respective job before throwing out the Future time out exception,
this can help to terminate the job and its respective tasks promptly when Timeout-exception happens, this will also save the additional resources getting utilized even after timeout exception thrown from driver.

After fix In Spark web UI the jobs are getting failed once timeout error occurs.

How was this patch tested?

Manually
Before fix

scala> val df1 = spark.range(0,10000,1,10000).selectExpr("id%10000 as key1", "id as value1")
df1: org.apache.spark.sql.DataFrame = [key1: bigint, value1: bigint]

scala> val df2 = spark.range(0,10000,1,10000).selectExpr("id%10000 as key2", "id as value2")
df2: org.apache.spark.sql.DataFrame = [key2: bigint, value2: bigint]

scala> val inner = df1.join(df2,col("key1")===col("key2")).select(col("key1"),col("value2")).collect

Actual Result : Timeout exception thrown and still task will be running in background, in spark web ui also the task execution will be in progress and after execution the job status shown successful, please refer attachments for more details.

Web UI

After Fix:
Once timeout occurs the job will be cancelled and even in UI the job status displayed as failed.

Web UI

…ce broadcast timeout occurs ## What changes were proposed in this pull request? Currently even Broadcast thread is timed out, Jobs are not aborted and it will run in the baakground, as per current design the broadcast future will be submitting the job whose result needs to be broadcasted wiithin a particular time, when the broadcast timeout happens the jobs which are scheduled will not getting killed and it will continue running in background even though time out happens. As part of solution we shall get the jobs based on execution id from appstatus store and cancel the respective job before throwing out the Future time out exception, this can help to terminate the job promptly when TimeOutException happens, this will also save the additional resources getting utilized even after timeout exception thrown from driver. In UI also the jobs are getting failed after applying this patch. ## How was this patch tested? Manually

sujith71955 · 2019-03-09T15:28:25Z

cc @srowen @HyukjinKwon @jinxing64
Please help to review this patch. Thanks

srowen · 2019-03-09T15:50:21Z

Is it necessary to kill all these jobs? I get that they will probably fail without the broadcast, but, it's also possible they won't.

sujith71955 · 2019-03-09T18:13:34Z

Is it necessary to kill all these jobs? I get that they will probably fail without the broadcast, but, it's also possible they won't.
You are right, the subsequent job will always failed without the broadcast results , and few points i want to highlight here is,
a) Broadcast job which is running in the background may unnecessarily occupy the resources , even-though the query will fail in the end , so it will be better if we can cancel these useless jobs/tasks which we are not sure how much time it can run.
b) Another point is regarding web ui, In UI broadcast job will be shown as completed which may simply mislead the users.

sujith71955 · 2019-03-09T18:40:01Z

@srowen hope i clarified your question. thanks

srowen · 2019-03-09T18:54:18Z

I don't feel strongly about it, but I am not sure it's worth this complexity.

sujith71955 · 2019-03-09T19:01:57Z

I don't feel strongly about it, but I am not sure it's worth this complexity.

For short tasks it may not harm much but long running tasks can unnecessary hold the resources, and ultimately the query results in failure. Moreover we dont have any control on type of broadcast jobs, it may vary based on business use-cases.

When the jobs are not able finish in 5 mins(Default Broadcast timeout time) which means it can turn out to be a long running job.

ajithme · 2019-03-11T04:08:47Z

@sujith71955 Few Questions,

In your scenario when the broadcast has timeout, will the query fail.?
Even if the query has failed, the broadcast tasks are running,?

sujith71955 · 2019-03-11T04:13:50Z

@ajithme

Yes, the query will fail as it will not get any result for broadcast once timeout occur.
broadcast jobs tasks will be running in the background because future timeout wont ensure termination of the tasks. since the task may run for long time also, it will unnecesarily occupy the resources.

ajithme · 2019-03-11T04:17:43Z

Is it necessary to kill all these jobs? I get that they will probably fail without the broadcast, but, it's also possible they won't.

@srowen then in this case how do we justify to the end user who will see that his query has failed (due to timeout) but his resource are still being occupied by the failed query (which may or may not eventually complete).

I think killing such orphaned tasks justifies the case

HyukjinKwon · 2019-03-11T09:42:15Z

ok to test

SparkQA · 2019-03-11T13:46:33Z

Test build #103314 has finished for PR 24036 at commit 43bfd81.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sujith71955 · 2019-03-13T03:52:06Z

@HyukjinKwon Any suggestions regarding this PR, please let me know for any inputs. thanks

sujith71955 · 2019-03-14T02:38:34Z

@HyukjinKwon @cloud-fan @srowen
Please let me know any further inputs.thnks

cloud-fan · 2019-03-14T11:25:16Z

sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala

    } catch {
      case ex: TimeoutException =>
        logError(s"Could not execute broadcast in ${timeout.toSeconds} secs.", ex)
+        val executionUIData = sqlContext.sparkSession.sharedState.statusStore.


Is this a reliable way to get the associated jobs?

SQLAppStatusListener will hold the live execution data, so i use the same for getting the associated jobs , If its not efficient way of getting then i will revisit the code and try to find a better mechanism for getting the associated job for the particular execution Id. please let me know for any suggestions. thanks for your valuable time.

I cannot find an alternative to get Jobs based on the execution id in this layer of execution, seems to be this way shall be reliable as whenever we are submitting /processing events via dagscheduler, we are always posting the events to SQLAppStatusListener, this will make our job viable to retrieve the jobs from LiveExecutionData.
Please let me know if we have any better way to get this job done. Thanks .

sujith71955 · 2019-03-18T06:05:19Z

Gentle ping @HyukjinKwon @cloud-fan @srowen

cloud-fan · 2019-03-18T10:05:44Z

maybe this is not the right place to do it. We should have a query manager that watches all the broadcasting of a query and cancel the entire query if one broadcasting fails.

sujith71955 · 2019-03-18T10:10:54Z

Yeah this shall be a cleaner approach, Need to think on the design 😊. Will workout on this part and let you guys know before implementation. Hope it’s fine

…

On Mon, 18 Mar 2019 at 3:36 PM, Wenchen Fan ***@***.***> wrote: maybe this is not the right place to do it. We should have a query manager that watches all the broadcasting of a query and cancel the entire query if one broadcasting fails. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24036 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMZZ-T5lI28gzrs3l9msto9JA-LLS0K4ks5vX2UzgaJpZM4bmirB> .

AmplabJenkins · 2019-03-29T19:39:30Z

Can one of the admins verify this patch?

sujith71955 · 2019-05-16T07:43:07Z

closing this PR as this scenario is already handled in below PR
https://github.com/apache/spark/pull/24595/files
Thanks all for your valuable inputs and time.

cloud-fan reviewed Mar 14, 2019

View reviewed changes

cloud-fan mentioned this pull request May 14, 2019

[SPARK-20774][SPARK-27036][SQL] Cancel the running broadcast execution on BroadcastTimeout #24595

Closed

sujith71955 closed this May 16, 2019

[SPARK-27036][SPARK-SQL] Cancel the running jobs in the background if broadcast future timeout error occurs #24036

[SPARK-27036][SPARK-SQL] Cancel the running jobs in the background if broadcast future timeout error occurs #24036

Uh oh!

Conversation

sujith71955 commented Mar 9, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

sujith71955 commented Mar 9, 2019

Uh oh!

srowen commented Mar 9, 2019

Uh oh!

sujith71955 commented Mar 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sujith71955 commented Mar 9, 2019

Uh oh!

srowen commented Mar 9, 2019

Uh oh!

sujith71955 commented Mar 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ajithme commented Mar 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sujith71955 commented Mar 11, 2019

Uh oh!

ajithme commented Mar 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Mar 11, 2019

Uh oh!

SparkQA commented Mar 11, 2019

Uh oh!

sujith71955 commented Mar 13, 2019

Uh oh!

sujith71955 commented Mar 14, 2019

Uh oh!

cloud-fan Mar 14, 2019

Choose a reason for hiding this comment

Uh oh!

sujith71955 Mar 14, 2019

Choose a reason for hiding this comment

Uh oh!

sujith71955 Mar 17, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sujith71955 commented Mar 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloud-fan commented Mar 18, 2019

Uh oh!

sujith71955 commented Mar 18, 2019 via email

Uh oh!

AmplabJenkins commented Mar 29, 2019

Uh oh!

sujith71955 commented May 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sujith71955 commented Mar 9, 2019 •

edited

Loading

sujith71955 commented Mar 9, 2019 •

edited

Loading

ajithme commented Mar 11, 2019 •

edited

Loading

ajithme commented Mar 11, 2019 •

edited

Loading

sujith71955 Mar 17, 2019 •

edited

Loading

sujith71955 commented Mar 18, 2019 •

edited

Loading