Skip to content

Commit 6c628e7

Browse files
gatorsmilecloud-fan
authored andcommitted
[MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and spark.sql.columnNameOfCorruptRecord
### What changes were proposed in this pull request? 1. The description of `spark.sql.files.ignoreCorruptFiles` is not accurate. When the file does not exist, we will issue the error message. ``` org.apache.spark.sql.AnalysisException: Path does not exist: file:/nonexist/path; ``` 2. `spark.sql.columnNameOfCorruptRecord` also affects the CSV format. The current description only mentions JSON format. ### How was this patch tested? N/A Author: Xiao Li <[email protected]> Closes #18184 from gatorsmile/updateMessage. (cherry picked from commit 2a780ac) Signed-off-by: Wenchen Fan <[email protected]>
1 parent 0c42279 commit 6c628e7

File tree

1 file changed

+3
-3
lines changed
  • sql/catalyst/src/main/scala/org/apache/spark/sql/internal

1 file changed

+3
-3
lines changed

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -345,7 +345,8 @@ object SQLConf {
345345
.createWithDefault(true)
346346

347347
val COLUMN_NAME_OF_CORRUPT_RECORD = buildConf("spark.sql.columnNameOfCorruptRecord")
348-
.doc("The name of internal column for storing raw/un-parsed JSON records that fail to parse.")
348+
.doc("The name of internal column for storing raw/un-parsed JSON and CSV records that fail " +
349+
"to parse.")
349350
.stringConf
350351
.createWithDefault("_corrupt_record")
351352

@@ -535,8 +536,7 @@ object SQLConf {
535536

536537
val IGNORE_CORRUPT_FILES = buildConf("spark.sql.files.ignoreCorruptFiles")
537538
.doc("Whether to ignore corrupt files. If true, the Spark jobs will continue to run when " +
538-
"encountering corrupted or non-existing and contents that have been read will still be " +
539-
"returned.")
539+
"encountering corrupted files and the contents that have been read will still be returned.")
540540
.booleanConf
541541
.createWithDefault(false)
542542

0 commit comments

Comments
 (0)