Skip to content
Snippets Groups Projects
Commit 2a780ac7 authored by Xiao Li's avatar Xiao Li Committed by Wenchen Fan
Browse files

[MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and...

[MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and spark.sql.columnNameOfCorruptRecord

### What changes were proposed in this pull request?
1. The description of `spark.sql.files.ignoreCorruptFiles` is not accurate. When the file does not exist, we will issue the error message.
```
org.apache.spark.sql.AnalysisException: Path does not exist: file:/nonexist/path;
```

2. `spark.sql.columnNameOfCorruptRecord` also affects the CSV format. The current description only mentions JSON format.

### How was this patch tested?
N/A

Author: Xiao Li <gatorsmile@gmail.com>

Closes #18184 from gatorsmile/updateMessage.
parent 16186cdc
No related branches found
No related tags found
No related merge requests found
......@@ -345,7 +345,8 @@ object SQLConf {
.createWithDefault(true)
val COLUMN_NAME_OF_CORRUPT_RECORD = buildConf("spark.sql.columnNameOfCorruptRecord")
.doc("The name of internal column for storing raw/un-parsed JSON records that fail to parse.")
.doc("The name of internal column for storing raw/un-parsed JSON and CSV records that fail " +
"to parse.")
.stringConf
.createWithDefault("_corrupt_record")
......@@ -535,8 +536,7 @@ object SQLConf {
val IGNORE_CORRUPT_FILES = buildConf("spark.sql.files.ignoreCorruptFiles")
.doc("Whether to ignore corrupt files. If true, the Spark jobs will continue to run when " +
"encountering corrupted or non-existing and contents that have been read will still be " +
"returned.")
"encountering corrupted files and the contents that have been read will still be returned.")
.booleanConf
.createWithDefault(false)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment