-
- Downloads
[SPARK-13255] [SQL] Update vectorized reader to directly return ColumnarBatch...
[SPARK-13255] [SQL] Update vectorized reader to directly return ColumnarBatch instead of InternalRows. ## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) Currently, the parquet reader returns rows one by one which is bad for performance. This patch updates the reader to directly return ColumnarBatches. This is only enabled with whole stage codegen, which is the only operator currently that is able to consume ColumnarBatches (instead of rows). The current implementation is a bit of a hack to get this to work and we should do more refactoring of these low level interfaces to make this work better. ## How was this patch tested? ``` Results: TPCDS: Best/Avg Time(ms) Rate(M/s) Per Row(ns) --------------------------------------------------------------------------------- q55 (before) 8897 / 9265 12.9 77.2 q55 5486 / 5753 21.0 47.6 ``` Author: Nong Li <nong@databricks.com> Closes #11435 from nongli/spark-13255.
Showing
- sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/UnsafeRowParquetRecordReader.java 24 additions, 5 deletions...ion/datasources/parquet/UnsafeRowParquetRecordReader.java
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java 57 additions, 0 deletions...che/spark/sql/execution/vectorized/ColumnVectorUtils.java
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarBatch.java 12 additions, 0 deletions.../apache/spark/sql/execution/vectorized/ColumnarBatch.java
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java 0 additions, 3 deletions...he/spark/sql/execution/vectorized/OnHeapColumnVector.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala 60 additions, 7 deletions...in/scala/org/apache/spark/sql/execution/ExistingRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala 65 additions, 7 deletions.../spark/sql/execution/datasources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SqlNewHadoopRDD.scala 7 additions, 1 deletion...che/spark/sql/execution/datasources/SqlNewHadoopRDD.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala 4 additions, 4 deletions...rk/sql/execution/datasources/parquet/ParquetIOSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadBenchmark.scala 55 additions, 18 deletions.../execution/datasources/parquet/ParquetReadBenchmark.scala
Loading
Please register or sign in to comment