-
- Downloads
[SPARK-17912] [SQL] Refactor code generation to get data for ColumnVector/ColumnarBatch
## What changes were proposed in this pull request? This PR refactors the code generation part to get data from `ColumnarVector` and `ColumnarBatch` by using a trait `ColumnarBatchScan` for ease of reuse. This is because this part will be reused by several components (e.g. parquet reader, Dataset.cache, and others) since `ColumnarBatch` will be first citizen. This PR is a part of https://github.com/apache/spark/pull/15219. In advance, this PR makes the code generation for `ColumnarVector` and `ColumnarBatch` reuseable as a trait. In general, this is very useful for other components from the reuseability view, too. ## How was this patch tested? tested existing test suites Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com> Closes #15467 from kiszk/columnarrefactor.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/ColumnarBatchScan.scala 133 additions, 0 deletions...la/org/apache/spark/sql/execution/ColumnarBatchScan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala 2 additions, 84 deletions...a/org/apache/spark/sql/execution/DataSourceScanExec.scala
Loading
Please register or sign in to comment