-
- Downloads
[SPARK-15493][SQL] default QuoteEscapingEnabled flag to true when writing CSV
## What changes were proposed in this pull request? Default QuoteEscapingEnabled flag to true when writing CSV and add an escapeQuotes option to be able to change this. See https://github.com/uniVocity/univocity-parsers/blob/f3eb2af26374940e60d91d1703bde54619f50c51/src/main/java/com/univocity/parsers/csv/CsvWriterSettings.java#L231-L247 This change is needed to be able to write RFC 4180 compatible CSV files (https://tools.ietf.org/html/rfc4180#section-2) https://issues.apache.org/jira/browse/SPARK-15493 ## How was this patch tested? Added a test that verifies the output is quoted correctly. Author: Jurriaan Pruis <email@jurriaanpruis.nl> Closes #13267 from jurriaan/quote-escaping.
Showing
- dev/deps/spark-deps-hadoop-2.2 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.2
- dev/deps/spark-deps-hadoop-2.3 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.3
- dev/deps/spark-deps-hadoop-2.4 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.4
- dev/deps/spark-deps-hadoop-2.6 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.6
- dev/deps/spark-deps-hadoop-2.7 1 addition, 1 deletiondev/deps/spark-deps-hadoop-2.7
- python/pyspark/sql/readwriter.py 6 additions, 1 deletionpython/pyspark/sql/readwriter.py
- sql/core/pom.xml 1 addition, 1 deletionsql/core/pom.xml
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 3 additions, 0 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala 2 additions, 0 deletions...ache/spark/sql/execution/datasources/csv/CSVOptions.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVParser.scala 1 addition, 0 deletions...pache/spark/sql/execution/datasources/csv/CSVParser.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala 51 additions, 0 deletions...apache/spark/sql/execution/datasources/csv/CSVSuite.scala
Loading
Please register or sign in to comment