-
- Downloads
[SPARK-18702][SQL] input_file_block_start and input_file_block_length
## What changes were proposed in this pull request? We currently have function input_file_name to get the path of the input file, but don't have functions to get the block start offset and length. This patch introduces two functions: 1. input_file_block_start: returns the file block start offset, or -1 if not available. 2. input_file_block_length: returns the file block length, or -1 if not available. ## How was this patch tested? Updated existing test cases in ColumnExpressionSuite that covered input_file_name to also cover the two new functions. Author: Reynold Xin <rxin@databricks.com> Closes #16133 from rxin/SPARK-18702.
Showing
- core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala 18 additions, 16 deletionscore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala
- core/src/main/scala/org/apache/spark/rdd/InputFileBlockHolder.scala 77 additions, 0 deletions...ain/scala/org/apache/spark/rdd/InputFileBlockHolder.scala
- core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala 23 additions, 20 deletionscore/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 2 additions, 0 deletions...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/inputFileBlock.scala 94 additions, 0 deletions...pache/spark/sql/catalyst/expressions/inputFileBlock.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 1 addition, 1 deletion...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala 5 additions, 4 deletions.../apache/spark/sql/execution/datasources/FileScanRDD.scala
- sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala 48 additions, 13 deletions...st/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
Loading
Please register or sign in to comment