[SPARK-5118][SQL] Fix: create table test stored as parquet as select .. #3921

guowei2 · 2015-01-07T03:14:03Z

No description provided.

AmplabJenkins · 2015-01-07T03:17:10Z

Can one of the admins verify this patch?

marmbrus · 2015-01-07T23:28:17Z

Can you add a test case? Perhaps in HiveQuerySuite

guowei2 · 2015-01-08T02:37:45Z

@marmbrus test case done

yhuai · 2015-01-09T00:49:07Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala

I guess you want to test STORED AS PARQUET?

guowei2 · 2015-01-09T03:53:37Z

i think i should remove the test case for stored as parquet only can pass in hive-0.13

marmbrus · 2015-01-09T04:27:22Z

Just check the version in the hive shim in the test case.
On Jan 8, 2015 7:53 PM, "guowei2" [email protected] wrote:

i think i should remove the test case for stored as parquet only can pass
in hive-0.13

—
Reply to this email directly or view it on GitHub
#3921 (comment).

yhuai · 2015-01-09T04:35:12Z

Like this one.

guowei2 · 2015-01-09T06:03:07Z

check the version in the hive shim in the test case Done
@yhuai thanks a lot

marmbrus · 2015-01-10T22:22:01Z

ok to test

marmbrus · 2015-01-10T22:22:19Z

Also can you please fix the merge conflict?

SparkQA · 2015-01-10T22:22:41Z

Test build #25364 has started for PR 3921 at commit 833f438.

This patch does not merge cleanly.

SparkQA · 2015-01-11T00:13:50Z

Test build #25364 has finished for PR 3921 at commit 833f438.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-11T00:13:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25364/
Test PASSed.

guowei2 · 2015-01-12T01:51:09Z

fix the merge conflict

SparkQA · 2015-01-12T01:52:31Z

Test build #25388 has started for PR 3921 at commit 5405b84.

This patch merges cleanly.

SparkQA · 2015-01-12T02:59:46Z

Test build #25388 has finished for PR 3921 at commit 5405b84.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-12T02:59:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25388/
Test PASSed.

yhuai · 2015-01-21T05:51:38Z

@guowei2 Can you rebase it? Also, can we add a test to make sure we actually generate the correct Parquet file? Just reading back the data of created table cannot verify the file format. Thank you.

yhuai · 2015-01-21T05:52:25Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala

We usually first get the original value of the setting and set it back at the end of the test.

SparkQA · 2015-01-21T07:27:42Z

Test build #25879 has started for PR 3921 at commit 9da56f8.

This patch merges cleanly.

guowei2 · 2015-01-21T07:31:01Z

@yhuai How to check the correct Parquet file?

create a parquet table, and then insert into data
create table stored as parquet as select ..
and then checkAnswer with them
is it OK?

SparkQA · 2015-01-21T08:33:44Z

Test build #25879 has finished for PR 3921 at commit 9da56f8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-21T08:33:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25879/
Test PASSed.

yhuai · 2015-01-21T17:27:30Z

I think you can use describe formatted <table name> to get the information of a table. You should be able to find InputFormat and OutputFormat in the output. We can just double check the output of describe and then make sure the data is stored in the correct format.

SparkQA · 2015-01-22T06:32:44Z

Test build #25948 has started for PR 3921 at commit b1ba3be.

This patch merges cleanly.

guowei2 · 2015-01-22T06:34:40Z

@yhuai I use DESC EXTENDED to check the the stored file, Is it OK?

SparkQA · 2015-01-22T07:37:28Z

Test build #25948 has finished for PR 3921 at commit b1ba3be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-22T07:37:31Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25948/
Test PASSed.

Author: guowei2 <[email protected]> Closes #3921 from guowei2/SPARK-5118 and squashes the following commits: b1ba3be [guowei2] add table file check in test case 9da56f8 [guowei2] test case only run in Shim13 112a0b6 [guowei2] add test case 187c7d8 [guowei2] Fix: create table test stored as parquet as select .. (cherry picked from commit e0490e2) Signed-off-by: Michael Armbrust <[email protected]>

marmbrus · 2015-02-04T23:26:47Z

Thanks for fixing this! Merged to master and branch 1.3.

guowei2 force-pushed the SPARK-5118 branch from 94fd9e3 to 5b3c4e3 Compare January 8, 2015 03:40

yhuai reviewed Jan 9, 2015
View reviewed changes

guowei2 force-pushed the SPARK-5118 branch from 833f438 to 5405b84 Compare January 12, 2015 01:49

yhuai reviewed Jan 21, 2015
View reviewed changes

guowei2 added 3 commits January 21, 2015 15:06

Fix: create table test stored as parquet as select ..

187c7d8

add test case

112a0b6

test case only run in Shim13

9da56f8

guowei2 force-pushed the SPARK-5118 branch from 5405b84 to 9da56f8 Compare January 21, 2015 07:24

add table file check in test case

b1ba3be

asfgit closed this in e0490e2 Feb 4, 2015

[SPARK-5118][SQL] Fix: create table test stored as parquet as select .. #3921

[SPARK-5118][SQL] Fix: create table test stored as parquet as select .. #3921

Uh oh!

Conversation

guowei2 commented Jan 7, 2015

Uh oh!

AmplabJenkins commented Jan 7, 2015

Uh oh!

marmbrus commented Jan 7, 2015

Uh oh!

guowei2 commented Jan 8, 2015

Uh oh!

yhuai Jan 9, 2015

Choose a reason for hiding this comment

Uh oh!

guowei2 commented Jan 9, 2015

Uh oh!

marmbrus commented Jan 9, 2015

Uh oh!

yhuai commented Jan 9, 2015

Uh oh!

guowei2 commented Jan 9, 2015

Uh oh!

marmbrus commented Jan 10, 2015

Uh oh!

marmbrus commented Jan 10, 2015

Uh oh!

SparkQA commented Jan 10, 2015

Uh oh!

SparkQA commented Jan 11, 2015

Uh oh!

AmplabJenkins commented Jan 11, 2015

Uh oh!

guowei2 commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

AmplabJenkins commented Jan 12, 2015

Uh oh!

yhuai commented Jan 21, 2015

Uh oh!

yhuai Jan 21, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

guowei2 commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

AmplabJenkins commented Jan 21, 2015

Uh oh!

yhuai commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 22, 2015

Uh oh!

guowei2 commented Jan 22, 2015

Uh oh!

SparkQA commented Jan 22, 2015

Uh oh!

AmplabJenkins commented Jan 22, 2015

Uh oh!

marmbrus commented Feb 4, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants