-
- Downloads
[SPARK-8357] Fix unsafe memory leak on empty inputs in GeneratedAggregate
This patch fixes a managed memory leak in GeneratedAggregate. The leak occurs when the unsafe aggregation path is used to perform grouped aggregation on an empty input; in this case, GeneratedAggregate allocates an UnsafeFixedWidthAggregationMap that is never cleaned up because `next()` is never called on the aggregate result iterator. This patch fixes this by short-circuiting on empty inputs. This patch is an updated version of #6810. Closes #6810. Author: navis.ryu <navis@apache.org> Author: Josh Rosen <joshrosen@databricks.com> Closes #7560 from JoshRosen/SPARK-8357 and squashes the following commits: 3486ce4 [Josh Rosen] Some minor cleanup c649310 [Josh Rosen] Revert SparkPlan change: 3c7db0f [Josh Rosen] Merge remote-tracking branch 'origin/master' into SPARK-8357 adc8239 [Josh Rosen] Back out Projection changes. c5419b3 [navis.ryu] addressed comments 143e1ef [navis.ryu] fixed format & added test for CCE case 735972f [navis.ryu] used new conf apis 1a02a55 [navis.ryu] Rolled-back test-conf cleanup & fixed possible CCE & added more tests 51178e8 [navis.ryu] addressed comments 4d326b9 [navis.ryu] fixed test fails 15c5afc [navis.ryu] added a test as suggested by JoshRosen d396589 [navis.ryu] added comments 1b07556 [navis.ryu] [SPARK-8357] [SQL] Memory leakage on unsafe aggregation path with empty input
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala 13 additions, 1 deletion...a/org/apache/spark/sql/execution/GeneratedAggregate.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 9 additions, 0 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/AggregateSuite.scala 48 additions, 0 deletions...scala/org/apache/spark/sql/execution/AggregateSuite.scala
Loading
Please register or sign in to comment