Skip to content

[SPARK-7084] improve saveAsTable documentation #5654

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 5 commits into from
Closed

[SPARK-7084] improve saveAsTable documentation #5654

wants to merge 5 commits into from

Conversation

@phatak-dev
Copy link
Contributor

@phatak-dev phatak-dev commented Apr 23, 2015

No description provided.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 23, 2015

Can one of the admins verify this patch?

rxin
rxin reviewed Apr 23, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala Outdated
Copy link
Contributor

@rxin rxin Apr 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe say

"Also note that while this function can persist the table metadata into Hive's metastore, the table will NOT be accessible from Hive."

@phatak-dev
Copy link
Contributor Author

phatak-dev commented Apr 23, 2015

Updated.

@rxin
Copy link
Contributor

rxin commented Apr 23, 2015

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30856 has started for PR 5654 at commit 00bc819.

@SparkQA
Copy link

SparkQA commented Apr 23, 2015

Test build #30856 has finished for PR 5654 at commit 00bc819.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class FreqItemset(namedtuple("FreqItemset", ["items", "freq"])):
    • protected[sql] abstract class AtomicType extends DataType
    • abstract class NumericType extends AtomicType
    • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
    • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
    • class Encoder[T <: AtomicType](columnType: NativeColumnType[T]) extends compression.Encoder[T]
    • class Decoder[T <: AtomicType](buffer: ByteBuffer, columnType: NativeColumnType[T])
  • This patch does not change any dependencies.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 23, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30856/
Test PASSed.

rxin
rxin reviewed Apr 23, 2015
View reviewed changes
sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
Copy link
Contributor

@rxin rxin Apr 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are multiple saveAsTable functions. Do you mind updating them as well?

@phatak-dev
Copy link
Contributor Author

phatak-dev commented Apr 24, 2015

Added for other methods also.

@AmplabJenkins
Copy link

AmplabJenkins commented Apr 27, 2015

Can one of the admins verify this patch?

@rxin
Copy link
Contributor

rxin commented May 11, 2015

Jenkins, test this please.

@AmplabJenkins
Copy link

AmplabJenkins commented May 11, 2015

Merged build triggered.

@AmplabJenkins
Copy link

AmplabJenkins commented May 11, 2015

Merged build started.

@SparkQA
Copy link

SparkQA commented May 11, 2015

Test build #32428 has started for PR 5654 at commit 386f407.

@rxin
Copy link
Contributor

rxin commented May 11, 2015

LGTM. I will merge it after Jenkins comes back happy.

@SparkQA
Copy link

SparkQA commented May 11, 2015

Test build #32428 has finished for PR 5654 at commit 386f407.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

AmplabJenkins commented May 11, 2015

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

AmplabJenkins commented May 11, 2015

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32428/
Test PASSed.

@asfgit asfgit closed this in 57255dc May 12, 2015
asfgit pushed a commit that referenced this pull request May 12, 2015
@phatak-dev @rxin
[SPARK-7084] improve saveAsTable documentation
0dbfe16 
Author: madhukar <[email protected]>

Closes #5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] #5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] #5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation

(cherry picked from commit 57255dc)
Signed-off-by: Reynold Xin <[email protected]>
asfgit pushed a commit that referenced this pull request May 12, 2015
@phatak-dev @rxin
[SPARK-7084] improve saveAsTable documentation
8f99a49 
Author: madhukar <[email protected]>

Closes #5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] #5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] #5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation

(cherry picked from commit 57255dc)
Signed-off-by: Reynold Xin <[email protected]>
@rxin
Copy link
Contributor

rxin commented May 12, 2015

FYI I submitted a small patch on top of this to add a link to a jira ticket: #6067

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@phatak-dev @jeanlyn
[SPARK-7084] improve saveAsTable documentation
697c79d 
Author: madhukar <[email protected]>

Closes apache#5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] apache#5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] apache#5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@phatak-dev @jeanlyn
[SPARK-7084] improve saveAsTable documentation
841fa8b 
Author: madhukar <[email protected]>

Closes apache#5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] apache#5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] apache#5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation
@chiyingyunhua
Copy link

chiyingyunhua commented Jun 18, 2015

请问，我使用spark1.3.1的sparksql中dataframe的一个saveastable方法存了一个dataframe，但是用hive读的时候出错了Failed with exception java.io.IOException:java.io.IOException: hdfs://namenode71:8020/user/hive/warehouse/zz5/part-00000 not a SequenceFile。这个怎么破？1.2是可以的

nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@phatak-dev @nemccarthy
[SPARK-7084] improve saveAsTable documentation
d5f3162 
Author: madhukar <[email protected]>

Closes apache#5654 from phatak-dev/master and squashes the following commits:

386f407 [madhukar] apache#5654 updated for all the methods
2c997c5 [madhukar] Merge branch 'master' of https://github.com/apache/spark
00bc819 [madhukar] Merge branch 'master' of https://github.com/apache/spark
2a802c6 [madhukar] apache#5654 updated the doc according to comments
866e8df [madhukar] [SPARK-7084] improve saveAsTable documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@phatak-dev @AmplabJenkins @rxin @SparkQA @chiyingyunhua