-
- Downloads
[SPARK-12922][SPARKR][WIP] Implement gapply() on DataFrame in SparkR
## What changes were proposed in this pull request? gapply() applies an R function on groups grouped by one or more columns of a DataFrame, and returns a DataFrame. It is like GroupedDataSet.flatMapGroups() in the Dataset API. Please, let me know what do you think and if you have any ideas to improve it. Thank you! ## How was this patch tested? Unit tests. 1. Primitive test with different column types 2. Add a boolean column 3. Compute average by a group Author: Narine Kokhlikyan <narine.kokhlikyan@gmail.com> Author: NarineK <narine.kokhlikyan@us.ibm.com> Closes #12836 from NarineK/gapply2.
Showing
- R/pkg/NAMESPACE 1 addition, 0 deletionsR/pkg/NAMESPACE
- R/pkg/R/DataFrame.R 81 additions, 1 deletionR/pkg/R/DataFrame.R
- R/pkg/R/deserialize.R 30 additions, 0 deletionsR/pkg/R/deserialize.R
- R/pkg/R/generics.R 4 additions, 0 deletionsR/pkg/R/generics.R
- R/pkg/R/group.R 62 additions, 0 deletionsR/pkg/R/group.R
- R/pkg/inst/tests/testthat/test_sparkSQL.R 65 additions, 0 deletionsR/pkg/inst/tests/testthat/test_sparkSQL.R
- R/pkg/inst/worker/worker.R 90 additions, 48 deletionsR/pkg/inst/worker/worker.R
- core/src/main/scala/org/apache/spark/api/r/RRunner.scala 17 additions, 3 deletionscore/src/main/scala/org/apache/spark/api/r/RRunner.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala 49 additions, 0 deletions.../org/apache/spark/sql/catalyst/plans/logical/object.scala
- sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala 47 additions, 1 deletion...scala/org/apache/spark/sql/RelationalGroupedDataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala 18 additions, 8 deletions.../src/main/scala/org/apache/spark/sql/api/r/SQLUtils.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 3 additions, 0 deletions...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/objects.scala 71 additions, 1 deletion...c/main/scala/org/apache/spark/sql/execution/objects.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/r/MapPartitionsRWrapper.scala 2 additions, 3 deletions.../apache/spark/sql/execution/r/MapPartitionsRWrapper.scala
Loading
Please register or sign in to comment