[SPARK-5196][SQL] Support `comment` in Create Table Field DDL #3999

OopsOutOfMemory · 2015-01-12T06:29:45Z

Support comment in create a table field.
CREATE TEMPORARY TABLE people(name string comment "the name of a person")

OopsOutOfMemory · 2015-01-12T06:29:57Z

This is the better way to implement the comment syntax.
Original implementation #3991 is to add a field comment to Expression, Atrribute, AtrributeReference which involved modifing too much files.

AmplabJenkins · 2015-01-12T06:32:09Z

Can one of the admins verify this patch?

OopsOutOfMemory · 2015-01-12T06:33:42Z

Jenkins, test this please

OopsOutOfMemory · 2015-01-12T06:37:28Z

@yhuai /cc @marmbrus
Could you review it ? After this, I'd like describe table command #3935 to support display the comment field of each column.

OopsOutOfMemory · 2015-01-12T13:28:52Z

ping.

yhuai · 2015-01-12T16:30:20Z

ok to test

SparkQA · 2015-01-12T16:32:35Z

Test build #25421 has started for PR 3999 at commit 81b8431.

This patch merges cleanly.

yhuai · 2015-01-12T16:36:59Z

sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala

How about we do not define the metadata if a comment is not provided? I feel we are losing information if we use an empty string when a comment is not provided.

I think the test failure will be resolved once we do not create a metadata if there is no comment.

that's fine.

SparkQA · 2015-01-12T16:44:30Z

Test build #25421 has finished for PR 3999 at commit 81b8431.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-12T16:44:33Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25421/
Test FAILed.

OopsOutOfMemory · 2015-01-12T17:30:17Z

test this please

OopsOutOfMemory · 2015-01-12T17:30:39Z

@yhuai /cc @marmbrus
Updated code.

SparkQA · 2015-01-12T17:32:36Z

Test build #25424 has started for PR 3999 at commit c2b6f3f.

This patch merges cleanly.

SparkQA · 2015-01-12T18:39:04Z

Test build #25424 has finished for PR 3999 at commit c2b6f3f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-12T18:39:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25424/
Test PASSed.

SparkQA · 2015-01-13T02:32:43Z

Test build #25445 has started for PR 3999 at commit 92fa3eb.

This patch merges cleanly.

SparkQA · 2015-01-13T02:33:41Z

Test build #25445 has finished for PR 3999 at commit 92fa3eb.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-13T02:33:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25445/
Test FAILed.

SparkQA · 2015-01-13T02:47:37Z

Test build #25449 has started for PR 3999 at commit 1d0fbac.

This patch merges cleanly.

SparkQA · 2015-01-13T02:50:35Z

Test build #25449 has finished for PR 3999 at commit 1d0fbac.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-13T02:50:37Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25449/
Test FAILed.

SparkQA · 2015-01-13T02:57:36Z

Test build #25450 has started for PR 3999 at commit d2c4caf.

This patch merges cleanly.

SparkQA · 2015-01-13T04:05:52Z

Test build #25450 has finished for PR 3999 at commit d2c4caf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-13T04:05:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25450/
Test PASSed.

OopsOutOfMemory · 2015-01-13T04:24:54Z

@yhuai /cc @marmbrus
Does this look OK to you?

OopsOutOfMemory · 2015-01-13T11:10:52Z

ping

SparkQA · 2015-01-14T09:47:39Z

Test build #25538 has started for PR 3999 at commit d5cba72.

This patch merges cleanly.

SparkQA · 2015-01-14T09:50:51Z

Test build #25538 has finished for PR 3999 at commit d5cba72.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2015-01-21T04:52:15Z

sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala

I think it is better to use queryExecution.executedPlan since it is the final physical. Also, this line is kind of long. How about we break it as follows.

val planned = sql("SELECT * FROM student").queryExecution.executedPlan val comments = planned.schema.fields.map { ...

yhuai · 2015-01-21T04:52:38Z

Just one minor comment. Other than that, LGTM.

SparkQA · 2015-01-21T06:32:44Z

Test build #25876 has started for PR 3999 at commit de6daa7.

This patch merges cleanly.

SparkQA · 2015-01-21T07:40:49Z

Test build #25876 has finished for PR 3999 at commit de6daa7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-21T07:40:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25876/
Test PASSed.

OopsOutOfMemory · 2015-01-21T07:52:51Z

/cc @yhuai @marmbrus
Test Passed.
Is that OK with you?

yhuai · 2015-01-21T14:54:46Z

LGTM

SparkQA · 2015-01-22T06:47:46Z

Test build #25949 has started for PR 3999 at commit 1c71505.

This patch merges cleanly.

SparkQA · 2015-01-22T06:57:43Z

Test build #25950 has started for PR 3999 at commit d1cfb0f.

This patch merges cleanly.

SparkQA · 2015-01-22T07:53:11Z

Test build #25949 has finished for PR 3999 at commit 1c71505.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-22T07:53:14Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25949/
Test PASSed.

SparkQA · 2015-01-22T08:00:15Z

Test build #25950 has finished for PR 3999 at commit d1cfb0f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-22T08:00:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25950/
Test PASSed.

OopsOutOfMemory · 2015-01-28T04:15:19Z

ping @marmbrus @yhuai
I think this is ready to go.

marmbrus · 2015-01-29T23:38:34Z

sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala

Plain asserts should be avoided in tests as they give very little information when there is a failure. Instead use === or add a custom string that describes what went wrong as the second argument.

SparkQA · 2015-01-30T02:52:35Z

Test build #26369 has started for PR 3999 at commit 77bf089.

This patch merges cleanly.

SparkQA · 2015-01-30T03:02:32Z

Test build #26371 has started for PR 3999 at commit 39150d4.

This patch merges cleanly.

SparkQA · 2015-01-30T04:00:25Z

Test build #26369 has finished for PR 3999 at commit 77bf089.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-30T04:00:28Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26369/
Test PASSed.

SparkQA · 2015-01-30T04:14:47Z

Test build #26371 has finished for PR 3999 at commit 39150d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-30T04:14:50Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26371/
Test PASSed.

OopsOutOfMemory · 2015-01-31T03:50:13Z

@marmbrus @rxin
Is this ready to merge?

marmbrus · 2015-02-02T02:20:17Z

sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala

I will fix this up while merging, but I don't think it is useful to both use === and give a comment that doesn't include the actual answer. Ideally, when the test fails you want to see the answer that does not match the expected answer so you can figure out what went wrong without having to go and add println manually.

OopsOutOfMemory mentioned this pull request Jan 12, 2015

[SPARK-5196][SQL] Support comment field in StructField and in CREATE TABLE DDL #3991

Closed

yhuai reviewed Jan 12, 2015
View reviewed changes

yhuai reviewed Jan 21, 2015
View reviewed changes

OopsOutOfMemory force-pushed the meta_comment branch from 1c71505 to d1cfb0f Compare January 22, 2015 06:52

marmbrus reviewed Jan 29, 2015
View reviewed changes

add comment and refine test suite

39150d4

OopsOutOfMemory force-pushed the meta_comment branch from 77bf089 to 39150d4 Compare January 30, 2015 02:57

marmbrus reviewed Feb 2, 2015
View reviewed changes

asfgit closed this in 1b56f1d Feb 2, 2015

OopsOutOfMemory deleted the meta_comment branch February 4, 2015 08:40

[SPARK-5196][SQL] Support comment in Create Table Field DDL #3999

[SPARK-5196][SQL] Support comment in Create Table Field DDL #3999

Uh oh!

Conversation

OopsOutOfMemory commented Jan 12, 2015

Uh oh!

OopsOutOfMemory commented Jan 12, 2015

Uh oh!

AmplabJenkins commented Jan 12, 2015

Uh oh!

OopsOutOfMemory commented Jan 12, 2015

Uh oh!

OopsOutOfMemory commented Jan 12, 2015

Uh oh!

OopsOutOfMemory commented Jan 12, 2015

Uh oh!

yhuai commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

yhuai Jan 12, 2015

Choose a reason for hiding this comment

Uh oh!

yhuai Jan 12, 2015

Choose a reason for hiding this comment

Uh oh!

OopsOutOfMemory Jan 12, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

AmplabJenkins commented Jan 12, 2015

Uh oh!

OopsOutOfMemory commented Jan 12, 2015

Uh oh!

OopsOutOfMemory commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 12, 2015

Uh oh!

AmplabJenkins commented Jan 12, 2015

Uh oh!

SparkQA commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 13, 2015

Uh oh!

AmplabJenkins commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 13, 2015

Uh oh!

AmplabJenkins commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 13, 2015

Uh oh!

AmplabJenkins commented Jan 13, 2015

Uh oh!

OopsOutOfMemory commented Jan 13, 2015

Uh oh!

OopsOutOfMemory commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 14, 2015

Uh oh!

SparkQA commented Jan 14, 2015

Uh oh!

yhuai Jan 21, 2015

Choose a reason for hiding this comment

Uh oh!

yhuai commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

AmplabJenkins commented Jan 21, 2015

Uh oh!

[SPARK-5196][SQL] Support `comment` in Create Table Field DDL #3999

[SPARK-5196][SQL] Support `comment` in Create Table Field DDL #3999