-
- Downloads
[SPARK-21354][SQL] INPUT FILE related functions do not support more than one sources
### What changes were proposed in this pull request? The build-in functions `input_file_name`, `input_file_block_start`, `input_file_block_length` do not support more than one sources, like what Hive does. Currently, Spark does not block it and the outputs are ambiguous/non-deterministic. It could be from any side. ``` hive> select *, INPUT__FILE__NAME FROM t1, t2; FAILED: SemanticException Column INPUT__FILE__NAME Found in more than One Tables/Subqueries ``` This PR blocks it and issues an error. ### How was this patch tested? Added a test case Author: gatorsmile <gatorsmile@gmail.com> Closes #18580 from gatorsmile/inputFileName.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala 37 additions, 1 deletion...la/org/apache/spark/sql/execution/datasources/rules.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala 1 addition, 0 deletions...g/apache/spark/sql/internal/BaseSessionStateBuilder.scala
- sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala 57 additions, 0 deletions...st/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala 1 addition, 2 deletions.../org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala 11 additions, 10 deletions...a/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala
Loading
Please register or sign in to comment