Skip to content
  • Cheng Hao's avatar
    2c3cc764
    [SPARK-3501] [SQL] Fix the bug of Hive SimpleUDF creates unnecessary type cast · 2c3cc764
    Cheng Hao authored
    When do the query like:
    ```
    select datediff(cast(value as timestamp), cast('2002-03-21 00:00:00' as timestamp)) from src;
    ```
    SparkSQL will raise exception:
    ```
    [info] scala.MatchError: TimestampType (of class org.apache.spark.sql.catalyst.types.TimestampType$)
    [info] at org.apache.spark.sql.catalyst.expressions.Cast.castToTimestamp(Cast.scala:77)
    [info] at org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:251)
    [info] at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:247)
    [info] at org.apache.spark.sql.catalyst.expressions.Cast.eval(Cast.scala:263)
    [info] at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$5$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:217)
    [info] at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$5$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:210)
    [info] at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:144)
    [info] at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$2.apply(TreeNode.scala:180)
    [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    ```
    
    Author: Cheng Hao <hao.cheng@intel.com>
    
    Closes #2368 from chenghao-intel/cast_exception and squashes the following commits:
    
    5c9c3a5 [Cheng Hao] make more clear code
    49dfc50 [Cheng Hao] Add no-op for Cast and revert the position of SimplifyCasts
    b804abd [Cheng Hao] Add unit test to show the failure in identical data type casting
    330a5c8 [Cheng Hao] Update Code based on comments
    b834ed4 [Cheng Hao] Fix bug of HiveSimpleUDF with unnecessary type cast which cause exception in constant folding
    2c3cc764
    [SPARK-3501] [SQL] Fix the bug of Hive SimpleUDF creates unnecessary type cast
    Cheng Hao authored
    When do the query like:
    ```
    select datediff(cast(value as timestamp), cast('2002-03-21 00:00:00' as timestamp)) from src;
    ```
    SparkSQL will raise exception:
    ```
    [info] scala.MatchError: TimestampType (of class org.apache.spark.sql.catalyst.types.TimestampType$)
    [info] at org.apache.spark.sql.catalyst.expressions.Cast.castToTimestamp(Cast.scala:77)
    [info] at org.apache.spark.sql.catalyst.expressions.Cast.cast$lzycompute(Cast.scala:251)
    [info] at org.apache.spark.sql.catalyst.expressions.Cast.cast(Cast.scala:247)
    [info] at org.apache.spark.sql.catalyst.expressions.Cast.eval(Cast.scala:263)
    [info] at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$5$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:217)
    [info] at org.apache.spark.sql.catalyst.optimizer.ConstantFolding$$anonfun$apply$5$$anonfun$applyOrElse$2.applyOrElse(Optimizer.scala:210)
    [info] at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:144)
    [info] at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4$$anonfun$apply$2.apply(TreeNode.scala:180)
    [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    [info] at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
    ```
    
    Author: Cheng Hao <hao.cheng@intel.com>
    
    Closes #2368 from chenghao-intel/cast_exception and squashes the following commits:
    
    5c9c3a5 [Cheng Hao] make more clear code
    49dfc50 [Cheng Hao] Add no-op for Cast and revert the position of SimplifyCasts
    b804abd [Cheng Hao] Add unit test to show the failure in identical data type casting
    330a5c8 [Cheng Hao] Update Code based on comments
    b834ed4 [Cheng Hao] Fix bug of HiveSimpleUDF with unnecessary type cast which cause exception in constant folding
Loading