Skip to content

[SPARK-22279][SQL] Enable convertMetastoreOrc by default #21186

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

[SPARK-22279][SQL] Enable convertMetastoreOrc by default #21186

wants to merge 1 commit into from

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Apr 27, 2018 •
edited
Loading

What changes were proposed in this pull request?

We reverted spark.sql.hive.convertMetastoreOrc at #20536 because we should not ignore the table-specific compression conf. Now, it's resolved via SPARK-23355.

How was this patch tested?

Pass the Jenkins.

@SparkQA
Copy link

SparkQA commented Apr 27, 2018

Test build #89941 has finished for PR 21186 at commit 5383299.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented Apr 29, 2018 •
edited
Loading

@gatorsmile and @cloud-fan .
Could you review this PR? This is a first try after the reverting (#20536).
If we have more things to do for this, please let me know.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-22279][SPARK-24112] Enable convertMetastoreOrc and add convertMetastore.TableProperty conf [SPARK-22279][SPARK-24112] Enable convertMetastoreOrc and add convertMetastoreTableProperty conf May 1, 2018
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 1, 2018

Hi, @gatorsmile .
Do you think we need to split this into two separate PRs? If you want, I will split this.

  • SPARK-22279 Enable convertMetastoreOrc by default
  • SPARK-24112 Add spark.sql.hive.convertMetastoreTableProperty for backward compatiblility

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 1, 2018

Retest this please.

@SparkQA
Copy link

SparkQA commented May 1, 2018

Test build #89986 has finished for PR 21186 at commit 5383299.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 1, 2018

The failures are irrelevant to this PR.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 1, 2018

Retest this please.

@SparkQA
Copy link

SparkQA commented May 1, 2018

Test build #89998 has finished for PR 21186 at commit 5383299.

  • This patch fails SparkR unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented May 2, 2018

spark.sql.hive.convertMetastoreTableProperty looks unnecessary to me...

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 2, 2018

Ya. I also thought like that before, @cloud-fan .

Please consider an existing customer environment like the unit test cases. For some Parquet tables having table properties like TBLPROPERTIES (parquet.compression 'NONE'), it was ignored by default before Apache Spark 2.4. After upgrading cluster, Spark will write uncompressed file which is different from Apache Spark 2.3 and old.

Since this is a behavior change, we need to document it and had better provide options for this. We can remove this at Apache Spark 3.0.

@SparkQA
Copy link

SparkQA commented May 3, 2018

Test build #90149 has finished for PR 21186 at commit b9ed640.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed May 4, 2018
View reviewed changes
docs/sql-programming-guide.md Outdated
Copy link
Contributor

@cloud-fan cloud-fan May 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please polish the migration guide w.r.t. https://issues.apache.org/jira/browse/SPARK-24175

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun May 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

dongjoon-hyun
dongjoon-hyun commented May 4, 2018
View reviewed changes
docs/sql-programming-guide.md Outdated
Copy link
Member Author

@dongjoon-hyun dongjoon-hyun May 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan and @gatorsmile . I updated according to the guideline SPARK-24175.

@SparkQA
Copy link

SparkQA commented May 4, 2018

Test build #90213 has finished for PR 21186 at commit b746702.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 6, 2018

Retest this please.

@SparkQA
Copy link

SparkQA commented May 6, 2018

Test build #90263 has finished for PR 21186 at commit b746702.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 7, 2018 •
edited
Loading

I'll split this into two PRs in order to make it easy to review.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-22279][SPARK-24112] Enable convertMetastoreOrc and add convertMetastoreTableProperty conf [SPARK-22279][SQL] Enable convertMetastoreOrc by default May 7, 2018
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 7, 2018

To reduce the review scope, convertMetastoreTableProperty goes to #21259 .

@SparkQA
Copy link

SparkQA commented May 7, 2018

Test build #90332 has finished for PR 21186 at commit ddd6872.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented May 9, 2018

can you resolve the conflicts?

@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 9, 2018

Sure, it's rebased now.

@SparkQA
Copy link

SparkQA commented May 9, 2018

Test build #90417 has finished for PR 21186 at commit 10e8319.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan
Copy link
Contributor

cloud-fan commented May 10, 2018

thanks, merging to master!

@asfgit asfgit closed this in e3d4349 May 10, 2018
@dongjoon-hyun
Copy link
Member Author

dongjoon-hyun commented May 10, 2018

Thank you, @cloud-fan !

@dongjoon-hyun dongjoon-hyun deleted the SPARK-24112 branch May 10, 2018 15:44
otterc pushed a commit to linkedin/spark that referenced this pull request Mar 22, 2023
@dongjoon-hyun
[SPARK-22279][SQL] Enable convertMetastoreOrc by default
ac59c17 
We reverted `spark.sql.hive.convertMetastoreOrc` at apache#20536 because we should not ignore the table-specific compression conf. Now, it's resolved via [SPARK-23355](apache@8aa1d7b).

Pass the Jenkins.

Author: Dongjoon Hyun <[email protected]>

Closes apache#21186 from dongjoon-hyun/SPARK-24112.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fan cloud-fan cloud-fan left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dongjoon-hyun @SparkQA @cloud-fan