[SPARK-16044][SQL] Backport input_file_name() for data source based on NewHadoopRDD to branch 1.6 #13806

HyukjinKwon · 2016-06-21T10:54:29Z

What changes were proposed in this pull request?

This PR backports #13759.

(SqlNewHadoopRDDState was renamed to InputFileNameHolder and spark API does not exist in branch 1.6)

How was this patch tested?

Unit tests in ColumnExpressionSuite.

…ranch 1.6

SparkQA · 2016-06-21T11:05:02Z

Test build #60926 has finished for PR 13806 at commit 2a55091.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-21T11:08:21Z

retest this please

SparkQA · 2016-06-21T13:47:41Z

Test build #60927 has finished for PR 13806 at commit 2a55091.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-21T13:54:05Z

retest this please

SparkQA · 2016-06-21T16:25:23Z

Test build #60935 has finished for PR 13806 at commit 2a55091.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-21T20:25:56Z

Test build #3121 has finished for PR 13806 at commit 2a55091.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-21T20:48:49Z

Test build #3122 has finished for PR 13806 at commit 2a55091.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-21T21:01:58Z

Test build #3123 has finished for PR 13806 at commit 2a55091.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-21T23:24:58Z

It seems the error is not related with this change.

HyukjinKwon · 2016-06-22T00:50:34Z

I will re-run this test here and meanwhile I will try to build in my local and run the Python tests.

HyukjinKwon · 2016-06-22T00:50:46Z

retest this please

SparkQA · 2016-06-22T03:44:26Z

Test build #60990 has finished for PR 13806 at commit 2a55091.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-22T04:42:15Z

I guess it is about f4af6a8. I will look into this deeper.

…oot` module ending up failure of Python tests ## What changes were proposed in this pull request? This PR fixes incorrect checking for `root` module (meaning all tests). I realised that #13806 is being failed due to this one. The PR corrects two files in `sql` and `core`. Since it seems fixing `core` module triggers all tests by `root` value from `determine_modules_for_files`. So, `changed_modules` becomes as below: ``` ['root', 'sql'] ``` and `module.dependent_modules` becaomes as below: ``` ['pyspark-mllib', 'pyspark-ml', 'hive-thriftserver', 'sparkr', 'mllib', 'examples', 'pyspark-sql'] ``` Now, `modules_to_test` does not include `root` and this checking is skipped but then both `changed_modules` and `modules_to_test` are being merged after that. So, this includes `root` module to test. This ends up with failing with the message below (e.g. https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60990/consoleFull): ``` Error: unrecognized module 'root'. Supported modules: pyspark-core, pyspark-sql, pyspark-streaming, pyspark-ml, pyspark-mllib ``` ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes #13845 from HyukjinKwon/fix-build-1.6.

JoshRosen · 2016-06-28T00:41:34Z

Jenkins, retest this please.

…oot` module ending up failure of Python tests ## What changes were proposed in this pull request? This PR fixes incorrect checking for `root` module (meaning all tests). I realised that apache#13806 is being failed due to this one. The PR corrects two files in `sql` and `core`. Since it seems fixing `core` module triggers all tests by `root` value from `determine_modules_for_files`. So, `changed_modules` becomes as below: ``` ['root', 'sql'] ``` and `module.dependent_modules` becaomes as below: ``` ['pyspark-mllib', 'pyspark-ml', 'hive-thriftserver', 'sparkr', 'mllib', 'examples', 'pyspark-sql'] ``` Now, `modules_to_test` does not include `root` and this checking is skipped but then both `changed_modules` and `modules_to_test` are being merged after that. So, this includes `root` module to test. This ends up with failing with the message below (e.g. https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60990/consoleFull): ``` Error: unrecognized module 'root'. Supported modules: pyspark-core, pyspark-sql, pyspark-streaming, pyspark-ml, pyspark-mllib ``` ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#13845 from HyukjinKwon/fix-build-1.6. (cherry picked from commit 4a67541)

SparkQA · 2016-06-28T04:53:25Z

Test build #61336 has finished for PR 13806 at commit 2a55091.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-28T04:58:36Z

Test build #61337 has finished for PR 13806 at commit 5ade5a2.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-28T05:05:57Z

retest this please

SparkQA · 2016-06-28T05:18:01Z

Test build #61352 has finished for PR 13806 at commit 5ade5a2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-28T05:19:41Z

Hm... am I doing something wrong here?

HyukjinKwon · 2016-06-28T05:20:44Z

retest this please

SparkQA · 2016-06-28T05:32:43Z

Test build #61353 has finished for PR 13806 at commit 5ade5a2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-28T05:47:13Z

Again, I think the error message is not related with this change. I will retest this and meanwhile try to build in my local.

HyukjinKwon · 2016-06-28T05:47:18Z

retest this please

SparkQA · 2016-06-28T05:57:46Z

Test build #61356 has finished for PR 13806 at commit 5ade5a2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-28T08:07:02Z

Test build #61357 has finished for PR 13806 at commit 5ade5a2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-28T08:12:45Z

retest this please

SparkQA · 2016-06-28T08:19:12Z

Test build #61367 has finished for PR 13806 at commit 5ade5a2.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-28T08:20:29Z

retest this please

SparkQA · 2016-06-28T08:47:51Z

Test build #61368 has finished for PR 13806 at commit 5ade5a2.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-28T09:11:40Z

retest this please

SparkQA · 2016-06-28T09:28:24Z

Test build #61370 has finished for PR 13806 at commit 5ade5a2.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-28T11:00:21Z

retest this please

SparkQA · 2016-06-28T13:15:06Z

Test build #61375 has finished for PR 13806 at commit 5ade5a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-06-28T14:13:48Z

@JoshRosen finally I got passed!

rxin · 2016-06-29T20:11:09Z

Merging. Thanks!

Can you close this?

…n NewHadoopRDD to branch 1.6 ## What changes were proposed in this pull request? This PR backports #13759. (`SqlNewHadoopRDDState` was renamed to `InputFileNameHolder` and `spark` API does not exist in branch 1.6) ## How was this patch tested? Unit tests in `ColumnExpressionSuite`. Author: hyukjinkwon <[email protected]> Closes #13806 from HyukjinKwon/backport-SPARK-16044.

…n NewHadoopRDD to branch 1.6 ## What changes were proposed in this pull request? This PR backports apache#13759. (`SqlNewHadoopRDDState` was renamed to `InputFileNameHolder` and `spark` API does not exist in branch 1.6) ## How was this patch tested? Unit tests in `ColumnExpressionSuite`. Author: hyukjinkwon <[email protected]> Closes apache#13806 from HyukjinKwon/backport-SPARK-16044. (cherry picked from commit 1ac830a)

Backport input_file_name() for data source based on NewHadoopRDD to b…

2a55091

…ranch 1.6

This was referenced Jun 22, 2016

[SPARK-13023][PROJECT INFRA][BRANCH-1.6] Fix handling of root module in modules_to_test() #12743

Closed

[SPARK-13023][PROJECT INFRA][FOLLOWUP][BRANCH-1.6] Unable to check root module ending up failure of Python tests #13845

Closed

Fetch upstream

5ade5a2

HyukjinKwon closed this Jun 29, 2016

HyukjinKwon reopened this Jun 29, 2016

HyukjinKwon closed this Jun 29, 2016

HyukjinKwon deleted the backport-SPARK-16044 branch January 2, 2018 03:40

[SPARK-16044][SQL] Backport input_file_name() for data source based on NewHadoopRDD to branch 1.6 #13806

[SPARK-16044][SQL] Backport input_file_name() for data source based on NewHadoopRDD to branch 1.6 #13806

Uh oh!

Conversation

HyukjinKwon commented Jun 21, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jun 21, 2016

Uh oh!

HyukjinKwon commented Jun 21, 2016

Uh oh!

SparkQA commented Jun 21, 2016

Uh oh!

HyukjinKwon commented Jun 21, 2016

Uh oh!

SparkQA commented Jun 21, 2016

Uh oh!

SparkQA commented Jun 21, 2016

Uh oh!

SparkQA commented Jun 21, 2016

Uh oh!

SparkQA commented Jun 21, 2016

Uh oh!

HyukjinKwon commented Jun 21, 2016

Uh oh!

HyukjinKwon commented Jun 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Jun 22, 2016

Uh oh!

SparkQA commented Jun 22, 2016

Uh oh!

HyukjinKwon commented Jun 22, 2016

Uh oh!

JoshRosen commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

SparkQA commented Jun 28, 2016

Uh oh!

HyukjinKwon commented Jun 28, 2016

Uh oh!

rxin commented Jun 29, 2016

Uh oh!

Reviewers

HyukjinKwon commented Jun 22, 2016 •

edited

Loading