Skip to content
Snippets Groups Projects
Commit 2a780ac7 authored by Xiao Li's avatar Xiao Li Committed by Wenchen Fan
Browse files

[MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and...

[MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and spark.sql.columnNameOfCorruptRecord

### What changes were proposed in this pull request?
1. The description of `spark.sql.files.ignoreCorruptFiles` is not accurate. When the file does not exist, we will issue the error message.
```
org.apache.spark.sql.AnalysisException: Path does not exist: file:/nonexist/path;
```

2. `spark.sql.columnNameOfCorruptRecord` also affects the CSV format. The current description only mentions JSON format.

### How was this patch tested?
N/A

Author: Xiao Li <gatorsmile@gmail.com>

Closes #18184 from gatorsmile/updateMessage.
parent 16186cdc
No related branches found
No related tags found
No related merge requests found
...@@ -345,7 +345,8 @@ object SQLConf { ...@@ -345,7 +345,8 @@ object SQLConf {
.createWithDefault(true) .createWithDefault(true)
val COLUMN_NAME_OF_CORRUPT_RECORD = buildConf("spark.sql.columnNameOfCorruptRecord") val COLUMN_NAME_OF_CORRUPT_RECORD = buildConf("spark.sql.columnNameOfCorruptRecord")
.doc("The name of internal column for storing raw/un-parsed JSON records that fail to parse.") .doc("The name of internal column for storing raw/un-parsed JSON and CSV records that fail " +
"to parse.")
.stringConf .stringConf
.createWithDefault("_corrupt_record") .createWithDefault("_corrupt_record")
...@@ -535,8 +536,7 @@ object SQLConf { ...@@ -535,8 +536,7 @@ object SQLConf {
val IGNORE_CORRUPT_FILES = buildConf("spark.sql.files.ignoreCorruptFiles") val IGNORE_CORRUPT_FILES = buildConf("spark.sql.files.ignoreCorruptFiles")
.doc("Whether to ignore corrupt files. If true, the Spark jobs will continue to run when " + .doc("Whether to ignore corrupt files. If true, the Spark jobs will continue to run when " +
"encountering corrupted or non-existing and contents that have been read will still be " + "encountering corrupted files and the contents that have been read will still be returned.")
"returned.")
.booleanConf .booleanConf
.createWithDefault(false) .createWithDefault(false)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment