From 13a47586fe97664d5fde4adb8aa44579226e7593 Mon Sep 17 00:00:00 2001 From: turbofei Date: Wed, 12 Feb 2020 14:52:15 +0800 Subject: [PATCH 1/2] [SPARK-29542][FOLLOW-UP] Keep the description of spark.sql.files.* consistent with that of SQLConf. --- docs/sql-performance-tuning.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md index e289854c7acc7..2efe8a51508b5 100644 --- a/docs/sql-performance-tuning.md +++ b/docs/sql-performance-tuning.md @@ -67,6 +67,7 @@ that these options will be deprecated in future release as more optimizations ar 134217728 (128 MB) The maximum number of bytes to pack into a single partition when reading files. + This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. @@ -76,7 +77,9 @@ that these options will be deprecated in future release as more optimizations ar The estimated cost to open a file, measured by the number of bytes could be scanned in the same time. This is used when putting multiple files into a partition. It is better to over-estimated, then the partitions with small files will be faster than partitions with bigger files (which is - scheduled first). + scheduled first). This configuration is effective only when using file-based sources such as Parquet, + JSON and ORC. + From 25b3febbfd94342e6e451b4682ec30ec34ee9308 Mon Sep 17 00:00:00 2001 From: turbofei Date: Wed, 12 Feb 2020 14:58:54 +0800 Subject: [PATCH 2/2] remove blank line --- docs/sql-performance-tuning.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md index 2efe8a51508b5..5a86c0cc31e12 100644 --- a/docs/sql-performance-tuning.md +++ b/docs/sql-performance-tuning.md @@ -79,7 +79,6 @@ that these options will be deprecated in future release as more optimizations ar then the partitions with small files will be faster than partitions with bigger files (which is scheduled first). This configuration is effective only when using file-based sources such as Parquet, JSON and ORC. -