-
- Downloads
[SPARK-11307] Reduce memory consumption of OutputCommitCoordinator
OutputCommitCoordinator uses a map in a place where an array would suffice, increasing its memory consumption for result stages with millions of tasks. This patch replaces that map with an array. The only tricky part of this is reasoning about the range of possible array indexes in order to make sure that we never index out of bounds. Author: Josh Rosen <joshrosen@databricks.com> Closes #9274 from JoshRosen/SPARK-11307.
Showing
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 7 additions, 1 deletion.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala 26 additions, 14 deletions.../org/apache/spark/scheduler/OutputCommitCoordinator.scala
- core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala 1 addition, 1 deletion...apache/spark/scheduler/OutputCommitCoordinatorSuite.scala
Loading
Please register or sign in to comment