Skip to content

[SPARK-29542][SQL][DOC] Make the descriptions of spark.sql.files.* be clearly #26200

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 6 commits into from
Closed

[SPARK-29542][SQL][DOC] Make the descriptions of spark.sql.files.* be clearly #26200

wants to merge 6 commits into from

Conversation

@turboFei
Copy link
Member

@turboFei turboFei commented Oct 22, 2019 •
edited
Loading

What changes were proposed in this pull request?

As described in SPARK-29542 , the descriptions of spark.sql.files.* are confused.
In this PR, I make their descriptions be clearly.

Why are the changes needed?

It makes the descriptions of spark.sql.files.* be clearly.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing UT.

@turboFei
Copy link
Member Author

@wangyum

@turboFei
Copy link
Member Author

turboFei commented Oct 22, 2019 •
edited
Loading

Oh, In fact, all the parameters started with spark.sql.files are only effective on spark data source table.

Such as:

  1. spark.sql.files.openCostInBytes
  2. spark.sql.files.ignoreCorruptFiles
  3. spark.sql.files.ignoreMissingFiles
  4. spark.sql.files.maxRecordsPerFile

Is it conventional that files represents data source?

@wangyum
Copy link
Member

wangyum commented Oct 22, 2019

ok to test

@SparkQA
Copy link

SparkQA commented Oct 22, 2019

Test build #112442 has finished for PR 26200 at commit dd28b09.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@turboFei turboFei changed the title [SPARK-29542] Make the description of spark.sql.files.maxPartitionBytes be clearly [SPARK-29542] Make the description of spark.sql.files.* be clearly Oct 22, 2019
@turboFei turboFei changed the title [SPARK-29542] Make the description of spark.sql.files.* be clearly [SPARK-29542] Make the descriptions of spark.sql.files.* be clearly Oct 22, 2019
@SparkQA
Copy link

SparkQA commented Oct 22, 2019

Test build #112475 has finished for PR 26200 at commit 4a56b2e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 22, 2019

Test build #112473 has finished for PR 26200 at commit 7ecb7b6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HeartSaVioR
Copy link
Contributor

HeartSaVioR commented Oct 22, 2019 •
edited
Loading

Looks like the title of PR needs to be updated to contain at least [SQL]. I'm not sure modifying description of config in code should be labeled as [DOC] in the title of PR.

@turboFei
Copy link
Member Author

Looks like the title of PR needs to be updated to contain at least [SQL]. I'm not sure modifying description of config in code should be labeled as [DOC] in the title of PR.

Thanks.

@turboFei turboFei changed the title [SPARK-29542] Make the descriptions of spark.sql.files.* be clearly [SPARK-29542][SQL][DOC] Make the descriptions of spark.sql.files.* be clearly Oct 22, 2019
HyukjinKwon
HyukjinKwon reviewed Oct 23, 2019
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine except https://github.com/apache/spark/pull/26200/files#r337833143

@SparkQA
Copy link

SparkQA commented Oct 23, 2019

Test build #112502 has finished for PR 26200 at commit 1de48df.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Oct 23, 2019

Test build #112506 has finished for PR 26200 at commit 7e115d8.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum
Copy link
Member

wangyum commented Oct 23, 2019

retest this please

@SparkQA
Copy link

SparkQA commented Oct 23, 2019

Test build #112518 has finished for PR 26200 at commit 7e115d8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Oct 23, 2019 •
edited
Loading

Merged to master.

@turboFei turboFei mentioned this pull request Feb 12, 2020
Closed
HyukjinKwon pushed a commit that referenced this pull request Feb 12, 2020
@turboFei @HyukjinKwon
[SPARK-29542][FOLLOW-UP] Keep the description of spark.sql.files.* in…
7c5d7d7 
… tuning guide be consistent with that in SQLConf

### What changes were proposed in this pull request?
This pr is a follow up of #26200.

In this PR, I modify the description of spark.sql.files.* in sql-performance-tuning.md to keep consistent with that in SQLConf.

### Why are the changes needed?

To keep consistent with the description in SQLConf.

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
Existed UT.

Closes #27545 from turboFei/SPARK-29542-follow-up.

Authored-by: turbofei <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
(cherry picked from commit 8b18397)
Signed-off-by: HyukjinKwon <[email protected]>
HyukjinKwon pushed a commit that referenced this pull request Feb 12, 2020
@turboFei @HyukjinKwon
[SPARK-29542][FOLLOW-UP] Keep the description of spark.sql.files.* in…
8b18397 
… tuning guide be consistent with that in SQLConf

### What changes were proposed in this pull request?
This pr is a follow up of #26200.

In this PR, I modify the description of spark.sql.files.* in sql-performance-tuning.md to keep consistent with that in SQLConf.

### Why are the changes needed?

To keep consistent with the description in SQLConf.

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
Existed UT.

Closes #27545 from turboFei/SPARK-29542-follow-up.

Authored-by: turbofei <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
sjincho pushed a commit to sjincho/spark that referenced this pull request Apr 15, 2020
@turboFei
[SPARK-29542][FOLLOW-UP] Keep the description of spark.sql.files.* in…
41e761c 
… tuning guide be consistent with that in SQLConf

### What changes were proposed in this pull request?
This pr is a follow up of apache#26200.

In this PR, I modify the description of spark.sql.files.* in sql-performance-tuning.md to keep consistent with that in SQLConf.

### Why are the changes needed?

To keep consistent with the description in SQLConf.

### Does this PR introduce any user-facing change?
No.

### How was this patch tested?
Existed UT.

Closes apache#27545 from turboFei/SPARK-29542-follow-up.

Authored-by: turbofei <[email protected]>
Signed-off-by: HyukjinKwon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wangyum wangyum wangyum left review comments

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees

No one assigned

Labels

DOCUMENTATION

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@turboFei @wangyum @SparkQA @HeartSaVioR @HyukjinKwon @dongjoon-hyun