Skip to content
Snippets Groups Projects
  • Reynold Xin's avatar
    926a93e5
    [SPARK-14088][SQL] Some Dataset API touch-up · 926a93e5
    Reynold Xin authored
    ## What changes were proposed in this pull request?
    1. Deprecated unionAll. It is pretty confusing to have both "union" and "unionAll" when the two do the same thing in Spark but are different in SQL.
    2. Rename reduce in KeyValueGroupedDataset to reduceGroups so it is more consistent with rest of the functions in KeyValueGroupedDataset. Also makes it more obvious what "reduce" and "reduceGroups" mean. Previously it was confusing because it could be reducing a Dataset, or just reducing groups.
    3. Added a "name" function, which is more natural to name columns than "as" for non-SQL users.
    4. Remove "subtract" function since it is just an alias for "except".
    
    ## How was this patch tested?
    All changes should be covered by existing tests. Also added couple test cases to cover "name".
    
    Author: Reynold Xin <rxin@databricks.com>
    
    Closes #11908 from rxin/SPARK-14088.
    926a93e5
    History
    [SPARK-14088][SQL] Some Dataset API touch-up
    Reynold Xin authored
    ## What changes were proposed in this pull request?
    1. Deprecated unionAll. It is pretty confusing to have both "union" and "unionAll" when the two do the same thing in Spark but are different in SQL.
    2. Rename reduce in KeyValueGroupedDataset to reduceGroups so it is more consistent with rest of the functions in KeyValueGroupedDataset. Also makes it more obvious what "reduce" and "reduceGroups" mean. Previously it was confusing because it could be reducing a Dataset, or just reducing groups.
    3. Added a "name" function, which is more natural to name columns than "as" for non-SQL users.
    4. Remove "subtract" function since it is just an alias for "except".
    
    ## How was this patch tested?
    All changes should be covered by existing tests. Also added couple test cases to cover "name".
    
    Author: Reynold Xin <rxin@databricks.com>
    
    Closes #11908 from rxin/SPARK-14088.