-
- Downloads
[SPARK-13883][SQL] Parquet Implementation of FileFormat.buildReader
This PR add implements the new `buildReader` interface for the Parquet `FileFormat`. An simple implementation of `FileScanRDD` is also included. This code should be tested by the many existing tests for parquet. Author: Michael Armbrust <michael@databricks.com> Author: Sameer Agarwal <sameer@databricks.com> Author: Nong Li <nong@databricks.com> Closes #11709 from marmbrus/parquetReader.
Showing
- sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java 71 additions, 11 deletions...on/datasources/parquet/VectorizedParquetRecordReader.java
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java 13 additions, 0 deletions...g/apache/spark/sql/execution/vectorized/ColumnVector.java
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarBatch.java 1 addition, 1 deletion.../apache/spark/sql/execution/vectorized/ColumnarBatch.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 0 additions, 1 deletion...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala 38 additions, 6 deletions.../apache/spark/sql/execution/datasources/FileScanRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala 23 additions, 13 deletions.../spark/sql/execution/datasources/FileSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/RecordReaderIterator.scala 55 additions, 0 deletions...park/sql/execution/datasources/RecordReaderIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala 143 additions, 5 deletions...k/sql/execution/datasources/parquet/ParquetRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 7 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala 6 additions, 3 deletions.../main/scala/org/apache/spark/sql/sources/interfaces.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala 2 additions, 2 deletions...rk/sql/execution/datasources/parquet/ParquetIOSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala 6 additions, 4 deletions...c/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala 1 addition, 0 deletions...a/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
Loading
Please register or sign in to comment