[SPARK-33934][SQL] Add SparkFile's root dir to env property PATH #30973

AngersZhuuuu · 2020-12-30T09:50:41Z

What changes were proposed in this pull request?

In hive we always use

add file /path/to/script.py;
select transform(col1, col2, ..)
using 'script.py' as (col1, col2, ...)
from ...

Since in spark we wrapper script command with /bash/bin -c, in this case we will throw script.py command not found.

This pr add a SparkFile's root dir path to execution env property PATH, then sub-processor will find scrip.py as program under PATH.

Why are the changes needed?

Support SQL migration form Hive to Spark.

Does this PR introduce any user-facing change?

User can direct use script file name as program in script transform SQL.

add file /path/to/script.py;
select transform(col1, col2, ..)
using 'script.py' as (col1, col2, ...)
from ...

How was this patch tested?

UT

SparkQA · 2020-12-30T10:55:13Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38118/

SparkQA · 2020-12-30T11:24:33Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38118/

SparkQA · 2020-12-30T12:31:01Z

Test build #133528 has finished for PR 30973 at commit c39e3cf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-30T15:58:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38125/

SparkQA · 2020-12-30T16:10:29Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38128/

SparkQA · 2020-12-30T16:26:43Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38125/

SparkQA · 2020-12-30T16:44:18Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38128/

SparkQA · 2020-12-30T17:23:04Z

Test build #133535 has finished for PR 30973 at commit 525b51e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-31T09:30:03Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38153/

SparkQA · 2020-12-31T10:05:19Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38153/

SparkQA · 2020-12-31T13:11:34Z

Test build #133563 has finished for PR 30973 at commit 0295501.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala

SparkQA · 2021-01-03T02:36:49Z

Test build #133605 has started for PR 30973 at commit f406844.

SparkQA · 2021-01-03T03:24:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38194/

SparkQA · 2021-01-03T03:55:04Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38194/

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala

maropu · 2021-01-04T01:38:48Z

This new feature itself looks useful. Btw, are you planning to make a PR to add a dedicated SQL document page for the TRANSFORM-related functionality, @AngersZhuuuu ? I think we need to document how to use the TRANSFORM clause for users. If a jira ticket for it doesn't exist, could you file it first?

AngersZhuuuu · 2021-01-04T02:07:17Z

This new feature itself looks useful. Btw, are you planning to make a PR to add a dedicated SQL document page for the TRANSFORM-related functionality, @AngersZhuuuu ? I think we need to document how to use the TRANSFORM clause for users. If a jira ticket for it doesn't exist, could you file it first?

Yea, I have planed to do this since a lot change done. #29087 (comment)
I have file a ticket. https://issues.apache.org/jira/browse/SPARK-33976

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala

sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala

HyukjinKwon · 2021-01-04T04:34:49Z

LGTM otherwise

maropu · 2021-01-04T05:58:31Z

Please update the PR description, too. Looks fine otherwise.

AngersZhuuuu · 2021-01-04T06:28:08Z

Please update the PR description, too. Looks fine otherwise.

Done

AngersZhuuuu · 2021-01-04T06:31:22Z

retest this please

HyukjinKwon · 2021-01-04T06:46:20Z

Merged to master.

dongjoon-hyun · 2021-01-05T19:34:39Z

Hi, all.
This seems to add a very flaky test. Could you take a look at that, please?

dongjoon-hyun · 2021-01-05T19:35:45Z

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala

 package org.apache.spark.sql.execution

-import java.io.{BufferedReader, InputStream, InputStreamReader, OutputStream}
+import java.io._


@AngersZhuuuu . nit. Please enumerate all next time.

dongjoon-hyun · 2021-01-05T19:36:42Z

sql/core/src/test/resources/test_script.py

@@ -1,3 +1,5 @@
+#! /usr/bin/python


Maybe, /usr/bin/python3?

AngersZhuuuu · 2021-01-05T20:55:42Z

Hi, all.
This seems to add a very flaky test. Could you take a look at that, please?

https://github.com/apache/spark/runs/1648908829

https://github.com/apache/spark/runs/1651998318

https://github.com/apache/spark/runs/1650788791

Seems. same issue as last time. I will raise a fix. pr soon.

…condition to fix flaky test ### What changes were proposed in this pull request? Follow comment and fix. flaky test #30973 (comment). This flaky test is similar as #30896 Some task's failed with root cause but in driver may return error without root cause , change. UT to check with status exit code since different root cause's exit code is not same. ### Why are the changes needed? Fix flaky test ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existed UT Closes #31046 from AngersZhuuuu/SPARK-33934-FOLLOW-UP. Lead-authored-by: angerszhu <[email protected]> Co-authored-by: AngersZhuuuu <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

[SPARK-33934][SQL] Support user defined script command wrapper like hive

c39e3cf

AngersZhuuuu marked this pull request as draft December 30, 2020 09:50

github-actions bot added the SQL label Dec 30, 2020

update

70b69a5

AngersZhuuuu marked this pull request as ready for review December 30, 2020 15:04

Fix UT

525b51e

AngersZhuuuu changed the title ~~[SPARK-33934][SQL] Support user defined script command wrapper like hive~~ [SPARK-33934][SQL] Support user defined script command wrapper and handle cmd arg like hive Dec 30, 2020

AngersZhuuuu added 2 commits December 31, 2020 16:15

Update SQLQuerySuite.scala

4268880

Update SQLQuerySuite.scala

0295501

HyukjinKwon reviewed Jan 3, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala Show resolved Hide resolved

HyukjinKwon reviewed Jan 3, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Jan 3, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala Show resolved Hide resolved

Update BaseScriptTransformationExec.scala

f406844

follow comment

6d305f5

maropu reviewed Jan 4, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala Outdated Show resolved Hide resolved

maropu reviewed Jan 4, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala Outdated Show resolved Hide resolved

maropu reviewed Jan 4, 2021

View reviewed changes

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala Show resolved Hide resolved

follow comment

e9e6984

HyukjinKwon reviewed Jan 4, 2021

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Jan 4, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/execution/BaseScriptTransformationSuite.scala Outdated Show resolved Hide resolved

AngersZhuuuu changed the title ~~[SPARK-33934][SQL] Support user defined script command wrapper and handle cmd arg like hive~~ [SPARK-33934][SQL] Add SparkFile's root dir to env property PATH Jan 4, 2021

follow comment

91542f9

HyukjinKwon closed this in adac633 Jan 4, 2021

dongjoon-hyun reviewed Jan 5, 2021

View reviewed changes

sql/core/src/test/resources/test_script.py

@@ -1,3 +1,5 @@

#! /usr/bin/python

Copy link

Member

dongjoon-hyun Jan 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, /usr/bin/python3?

AngersZhuuuu mentioned this pull request Jan 5, 2021

[SPARK-33934][SQL][FOLLOW-UP] Use SubProcessor's exit code as assert condition to fix flaky test #31046

Closed

hammertank mentioned this pull request Feb 8, 2021

[SPARK-33152][SQL] Improve the performance of constraint propagation for Project and Aggregate #30894

Closed

[SPARK-33934][SQL] Add SparkFile's root dir to env property PATH #30973

[SPARK-33934][SQL] Add SparkFile's root dir to env property PATH #30973

Uh oh!

Conversation

AngersZhuuuu commented Dec 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Dec 30, 2020

Uh oh!

SparkQA commented Dec 30, 2020

Uh oh!

SparkQA commented Dec 30, 2020

Uh oh!

SparkQA commented Dec 30, 2020

Uh oh!

SparkQA commented Dec 30, 2020

Uh oh!

SparkQA commented Dec 30, 2020

Uh oh!

SparkQA commented Dec 30, 2020

Uh oh!

SparkQA commented Dec 30, 2020

Uh oh!

SparkQA commented Dec 31, 2020

Uh oh!

SparkQA commented Dec 31, 2020

Uh oh!

SparkQA commented Dec 31, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Jan 3, 2021

Uh oh!

SparkQA commented Jan 3, 2021

Uh oh!

SparkQA commented Jan 3, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maropu commented Jan 4, 2021

Uh oh!

AngersZhuuuu commented Jan 4, 2021

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Jan 4, 2021

Uh oh!

maropu commented Jan 4, 2021

Uh oh!

AngersZhuuuu commented Jan 4, 2021

Uh oh!

AngersZhuuuu commented Jan 4, 2021

Uh oh!

HyukjinKwon commented Jan 4, 2021

Uh oh!

dongjoon-hyun commented Jan 5, 2021

Uh oh!

dongjoon-hyun Jan 5, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jan 5, 2021

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu commented Jan 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AngersZhuuuu commented Dec 30, 2020 •

edited

Loading