-
- Downloads
[SPARK-10310] [SQL] Fixes script transformation field/line delimiters
**Please attribute this PR to `Zhichao Li <zhichao.liintel.com>`.** This PR is based on PR #8476 authored by zhichao-li. It fixes SPARK-10310 by adding field delimiter SerDe property to the default `LazySimpleSerDe`, and enabling default record reader/writer classes. Currently, we only support `LazySimpleSerDe`, used together with `TextRecordReader` and `TextRecordWriter`, and don't support customizing record reader/writer using `RECORDREADER`/`RECORDWRITER` clauses. This should be addressed in separate PR(s). Author: Cheng Lian <lian@databricks.com> Closes #8860 from liancheng/spark-10310/fix-script-trans-delimiters.
Showing
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 43 additions, 9 deletions...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala 62 additions, 13 deletions...pache/spark/sql/hive/execution/ScriptTransformation.scala
- sql/hive/src/test/resources/data/scripts/test_transform.py 6 additions, 0 deletionssql/hive/src/test/resources/data/scripts/test_transform.py
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 39 additions, 0 deletions...a/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ScriptTransformationSuite.scala 2 additions, 0 deletions.../spark/sql/hive/execution/ScriptTransformationSuite.scala
Loading
Please register or sign in to comment