Skip to content
  • Dongjoon Hyun's avatar
    46395db8
    [SPARK-16289][SQL] Implement posexplode table generating function · 46395db8
    Dongjoon Hyun authored
    ## What changes were proposed in this pull request?
    
    This PR implements `posexplode` table generating function. Currently, master branch raises the following exception for `map` argument. It's different from Hive.
    
    **Before**
    ```scala
    scala> sql("select posexplode(map('a', 1, 'b', 2))").show
    org.apache.spark.sql.AnalysisException: No handler for Hive UDF ... posexplode() takes an array as a parameter; line 1 pos 7
    ```
    
    **After**
    ```scala
    scala> sql("select posexplode(map('a', 1, 'b', 2))").show
    +---+---+-----+
    |pos|key|value|
    +---+---+-----+
    |  0|  a|    1|
    |  1|  b|    2|
    +---+---+-----+
    ```
    
    For `array` argument, `after` is the same with `before`.
    ```
    scala> sql("select posexplode(array(1, 2, 3))").show
    +---+---+
    |pos|col|
    +---+---+
    |  0|  1|
    |  1|  2|
    |  2|  3|
    +---+---+
    ```
    
    ## How was this patch tested?
    
    Pass the Jenkins tests with newly added testcases.
    
    Author: Dongjoon Hyun <dongjoon@apache.org>
    
    Closes #13971 from dongjoon-hyun/SPARK-16289.
    46395db8
    [SPARK-16289][SQL] Implement posexplode table generating function
    Dongjoon Hyun authored
    ## What changes were proposed in this pull request?
    
    This PR implements `posexplode` table generating function. Currently, master branch raises the following exception for `map` argument. It's different from Hive.
    
    **Before**
    ```scala
    scala> sql("select posexplode(map('a', 1, 'b', 2))").show
    org.apache.spark.sql.AnalysisException: No handler for Hive UDF ... posexplode() takes an array as a parameter; line 1 pos 7
    ```
    
    **After**
    ```scala
    scala> sql("select posexplode(map('a', 1, 'b', 2))").show
    +---+---+-----+
    |pos|key|value|
    +---+---+-----+
    |  0|  a|    1|
    |  1|  b|    2|
    +---+---+-----+
    ```
    
    For `array` argument, `after` is the same with `before`.
    ```
    scala> sql("select posexplode(array(1, 2, 3))").show
    +---+---+
    |pos|col|
    +---+---+
    |  0|  1|
    |  1|  2|
    |  2|  3|
    +---+---+
    ```
    
    ## How was this patch tested?
    
    Pass the Jenkins tests with newly added testcases.
    
    Author: Dongjoon Hyun <dongjoon@apache.org>
    
    Closes #13971 from dongjoon-hyun/SPARK-16289.
Loading