-
- Downloads
[SPARK-8005][SQL] Input file name
Users can now get the file name of the partition being read in. A thread local variable is in `SQLNewHadoopRDD` and is set when the partition is computed. `SQLNewHadoopRDD` is moved to core so that the catalyst package can reach it.
This supports:
`df.select(inputFileName())`
and
`sqlContext.sql("select input_file_name() from table")`
Author: Joseph Batchik <josephbatchik@gmail.com>
Closes #7743 from JDrit/input_file_name and squashes the following commits:
abb8609 [Joseph Batchik] fixed failing test and changed the default value to be an empty string
d2f323d [Joseph Batchik] updates per review
102061f [Joseph Batchik] updates per review
75313f5 [Joseph Batchik] small fixes
c7f7b5a [Joseph Batchik] addeding input file name to Spark SQL
Showing
- core/src/main/scala/org/apache/spark/rdd/SqlNewHadoopRDD.scala 29 additions, 5 deletions...src/main/scala/org/apache/spark/rdd/SqlNewHadoopRDD.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 2 additions, 1 deletion...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InputFileName.scala 49 additions, 0 deletions...apache/spark/sql/catalyst/expressions/InputFileName.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SparkPartitionID.scala 2 additions, 0 deletions...che/spark/sql/catalyst/expressions/SparkPartitionID.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/NondeterministicSuite.scala 4 additions, 0 deletions...park/sql/catalyst/expressions/NondeterministicSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 9 additions, 0 deletionssql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala 1 addition, 2 deletions.../scala/org/apache/spark/sql/parquet/ParquetRelation.scala
- sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala 16 additions, 1 deletion...st/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala 16 additions, 1 deletionsql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/UDFSuite.scala 0 additions, 6 deletions...e/src/test/scala/org/apache/spark/sql/hive/UDFSuite.scala
Loading
Please register or sign in to comment