-
- Downloads
[SPARK-5212][SQL] Add support of schema-less, custom field delimiter and SerDe for HiveQL transform
This pr adds the support of schema-less syntax, custom field delimiter and SerDe for HiveQL's transform. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #4014 from viirya/schema_less_trans and squashes the following commits: ac2d1fe [Liang-Chi Hsieh] Refactor codes for comments. a137933 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into schema_less_trans aa10fbd [Liang-Chi Hsieh] Add Hive golden answer files again. 575f695 [Liang-Chi Hsieh] Add Hive golden answer files for new unit tests. a422562 [Liang-Chi Hsieh] Use createQueryTest for unit tests and remove unnecessary imports. ccb71e3 [Liang-Chi Hsieh] Refactor codes for comments. 37bd391 [Liang-Chi Hsieh] Merge remote-tracking branch 'upstream/master' into schema_less_trans 6000889 [Liang-Chi Hsieh] Wrap input and output schema into ScriptInputOutputSchema. 21727f7 [Liang-Chi Hsieh] Move schema-less output to proper place. Use multilines instead of a long line SQL. 9a6dc04 [Liang-Chi Hsieh] setRecordReaderID is introduced in 0.13.1, use reflection API to call it. 7a14f31 [Liang-Chi Hsieh] Fix bug. 799b5e1 [Liang-Chi Hsieh] Call getSerializedClass instead of using Text. be2c3fc [Liang-Chi Hsieh] Fix style. 32d3046 [Liang-Chi Hsieh] Add SerDe support. ab22f7b [Liang-Chi Hsieh] Fix style. 7a48e42 [Liang-Chi Hsieh] Add support of custom field delimiter. b1729d9 [Liang-Chi Hsieh] Fix style. ccee49e [Liang-Chi Hsieh] Add unit test. f561c37 [Liang-Chi Hsieh] Add support of schema-less script transformation.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/ScriptTransformation.scala 9 additions, 1 deletion...ark/sql/catalyst/plans/logical/ScriptTransformation.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 47 additions, 12 deletions...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 2 additions, 2 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/ScriptTransformation.scala 205 additions, 15 deletions...pache/spark/sql/hive/execution/ScriptTransformation.scala
- sql/hive/src/test/resources/golden/schema-less transform-0-d5738de14dd6e29da712ec3318f4118f 500 additions, 0 deletions.../schema-less transform-0-d5738de14dd6e29da712ec3318f4118f
- sql/hive/src/test/resources/golden/schema-less transform-1-49624ef4e2c3cc2040c06660b926219b 500 additions, 0 deletions.../schema-less transform-1-49624ef4e2c3cc2040c06660b926219b
- sql/hive/src/test/resources/golden/transform with SerDe-0-cdc393f3914c879787efe523f692b1e0 500 additions, 0 deletions...n/transform with SerDe-0-cdc393f3914c879787efe523f692b1e0
- sql/hive/src/test/resources/golden/transform with SerDe3-0-58a8b7eb07a949bc44dccb723222957f 500 additions, 0 deletions.../transform with SerDe3-0-58a8b7eb07a949bc44dccb723222957f
- sql/hive/src/test/resources/golden/transform with SerDe4-0-ba9ad2499a7408cb350c7abafaf9ea97 500 additions, 0 deletions.../transform with SerDe4-0-ba9ad2499a7408cb350c7abafaf9ea97
- sql/hive/src/test/resources/golden/transform with custom field delimiter-0-703cca3c02ced422feb11dc13b744484 500 additions, 0 deletions...custom field delimiter-0-703cca3c02ced422feb11dc13b744484
- sql/hive/src/test/resources/golden/transform with custom field delimiter-0-82639dda9ba42df817466dffe2929174 500 additions, 0 deletions...custom field delimiter-0-82639dda9ba42df817466dffe2929174
- sql/hive/src/test/resources/golden/transform with custom field delimiter2-0-e8713b21483e1efb78ee90b61530479b 500 additions, 0 deletions...ustom field delimiter2-0-e8713b21483e1efb78ee90b61530479b
- sql/hive/src/test/resources/golden/transform with custom field delimiter2-0-e8d2b2e60551f69bfb44e555f5cff064 500 additions, 0 deletions...ustom field delimiter2-0-e8d2b2e60551f69bfb44e555f5cff064
- sql/hive/src/test/resources/golden/transform with custom field delimiter3-0-d4f4f471819345e9ce1964e281ea5289 500 additions, 0 deletions...ustom field delimiter3-0-d4f4f471819345e9ce1964e281ea5289
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveQuerySuite.scala 75 additions, 1 deletion.../org/apache/spark/sql/hive/execution/HiveQuerySuite.scala
- sql/hive/v0.12.0/src/main/scala/org/apache/spark/sql/hive/Shim12.scala 5 additions, 1 deletion...2.0/src/main/scala/org/apache/spark/sql/hive/Shim12.scala
- sql/hive/v0.13.1/src/main/scala/org/apache/spark/sql/hive/Shim13.scala 17 additions, 2 deletions...3.1/src/main/scala/org/apache/spark/sql/hive/Shim13.scala
Loading
Please register or sign in to comment