-
- Downloads
[SPARK-16610][SQL] Add `orc.compress` as an alias for `compression` option.
## What changes were proposed in this pull request? For ORC source, Spark SQL has a writer option `compression`, which is used to set the codec and its value will be also set to `orc.compress` (the orc conf used for codec). However, if a user only set `orc.compress` in the writer option, we should not use the default value of `compression` (snappy) as the codec. Instead, we should respect the value of `orc.compress`. This PR makes ORC data source not ignoring `orc.compress` when `comperssion` is unset. So, here is the behaviour, 1. Check `compression` and use this if it is set. 2. If `compression` is not set, check `orc.compress` and use it. 3. If `compression` and `orc.compress` are not set, then use the default snappy. ## How was this patch tested? Unit test in `OrcQuerySuite`. Author: hyukjinkwon <gurwls223@gmail.com> Closes #14518 from HyukjinKwon/SPARK-16610.
Showing
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala 9 additions, 3 deletions...main/scala/org/apache/spark/sql/hive/orc/OrcOptions.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala 23 additions, 0 deletions...t/scala/org/apache/spark/sql/hive/orc/OrcQuerySuite.scala
Loading
Please register or sign in to comment