This is an assembly module for Spark project. It creates a single tar.gz file that includes all needed dependency of the project except for org.apache.hadoop.* jars that are supposed to be available from the deployed Hadoop cluster. This module is off by default. To activate it specify the profile in the command line -Pbigtop-dist If you need to build an assembly for a different version of Hadoop the hadoop-version system property needs to be set as in this example: -Dhadoop.version=2.7.3
Dongjoon Hyun
authored
## What changes were proposed in this pull request? Like Parquet, this PR aims to depend on the latest Apache ORC 1.4 for Apache Spark 2.3. There are key benefits for Apache ORC 1.4. - Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more. - Maintainability: Reduce the Hive dependency and can remove old legacy code later. Later, we can get the following two key benefits by adding new ORCFileFormat in SPARK-20728 (#17980), too. - Usability: User can use ORC data sources without hive module, i.e, -Phive. - Speed: Use both Spark ColumnarBatch and ORC RowBatch together. This will be faster than the current implementation in Spark. ## How was this patch tested? Pass the jenkins. Author: Dongjoon Hyun <dongjoon@apache.org> Closes #18640 from dongjoon-hyun/SPARK-21422.
| Name | Last commit | Last update |
|---|---|---|
| .. | ||
| src/main/assembly | ||
| README | ||
| pom.xml | ||