Skip to content
Snippets Groups Projects
  • Michael Armbrust's avatar
    6d0633e3
    [SPARK-7548] [SQL] Add explode function for DataFrames · 6d0633e3
    Michael Armbrust authored
    Add an `explode` function for dataframes and modify the analyzer so that single table generating functions can be present in a select clause along with other expressions.   There are currently the following restrictions:
     - only top level TGFs are allowed (i.e. no `select(explode('list) + 1)`)
     - only one may be present in a single select to avoid potentially confusing implicit Cartesian products.
    
    TODO:
     - [ ] Python
    
    Author: Michael Armbrust <michael@databricks.com>
    
    Closes #6107 from marmbrus/explodeFunction and squashes the following commits:
    
    7ee2c87 [Michael Armbrust] whitespace
    6f80ba3 [Michael Armbrust] Update dataframe.py
    c176c89 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into explodeFunction
    81b5da3 [Michael Armbrust] style
    d3faa05 [Michael Armbrust] fix self join case
    f9e1e3e [Michael Armbrust] fix python, add since
    4f0d0a9 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into explodeFunction
    e710fe4 [Michael Armbrust] add java and python
    52ca0dc [Michael Armbrust] [SPARK-7548][SQL] Add explode function for dataframes.
    6d0633e3
    History
    [SPARK-7548] [SQL] Add explode function for DataFrames
    Michael Armbrust authored
    Add an `explode` function for dataframes and modify the analyzer so that single table generating functions can be present in a select clause along with other expressions.   There are currently the following restrictions:
     - only top level TGFs are allowed (i.e. no `select(explode('list) + 1)`)
     - only one may be present in a single select to avoid potentially confusing implicit Cartesian products.
    
    TODO:
     - [ ] Python
    
    Author: Michael Armbrust <michael@databricks.com>
    
    Closes #6107 from marmbrus/explodeFunction and squashes the following commits:
    
    7ee2c87 [Michael Armbrust] whitespace
    6f80ba3 [Michael Armbrust] Update dataframe.py
    c176c89 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into explodeFunction
    81b5da3 [Michael Armbrust] style
    d3faa05 [Michael Armbrust] fix self join case
    f9e1e3e [Michael Armbrust] fix python, add since
    4f0d0a9 [Michael Armbrust] Merge remote-tracking branch 'origin/master' into explodeFunction
    e710fe4 [Michael Armbrust] add java and python
    52ca0dc [Michael Armbrust] [SPARK-7548][SQL] Add explode function for dataframes.