-
- Downloads
"git@gitlab.engr.illinois.edu:croft1/floodlight.git" did not exist on "9b431bca7368c3219a175a6d2ae930965cc5694a"
[SPARK-9709] [SQL] Avoid starving unsafe operators that use sort
The issue is that a task may run multiple sorts, and the sorts run by the child operator (i.e. parent RDD) may acquire all available memory such that other sorts in the same task do not have enough to proceed. This manifests itself in an `IOException("Unable to acquire X bytes of memory")` thrown by `UnsafeExternalSorter`. The solution is to reserve a page in each sorter in the chain before computing the child operator's (parent RDD's) partitions. This requires us to use a new special RDD that does some preparation before computing the parent's partitions. Author: Andrew Or <andrew@databricks.com> Closes #8011 from andrewor14/unsafe-starve-memory and squashes the following commits: 35b69a4 [Andrew Or] Simplify test 0b07782 [Andrew Or] Minor: update comments 5d5afdf [Andrew Or] Merge branch 'master' of github.com:apache/spark into unsafe-starve-memory 254032e [Andrew Or] Add tests 234acbd [Andrew Or] Reserve a page in sorter when preparing each partition b889e08 [Andrew Or] MapPartitionsWithPreparationRDD
Showing
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 29 additions, 14 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala 3 additions, 0 deletions...rc/main/scala/org/apache/spark/rdd/MapPartitionsRDD.scala
- core/src/main/scala/org/apache/spark/rdd/MapPartitionsWithPreparationRDD.scala 49 additions, 0 deletions...rg/apache/spark/rdd/MapPartitionsWithPreparationRDD.scala
- core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala 1 addition, 1 deletion...scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala
- core/src/test/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorterSuite.java 18 additions, 1 deletion...til/collection/unsafe/sort/UnsafeExternalSorterSuite.java
- core/src/test/scala/org/apache/spark/rdd/MapPartitionsWithPreparationRDDSuite.scala 60 additions, 0 deletions...ache/spark/rdd/MapPartitionsWithPreparationRDDSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 1 addition, 1 deletion...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/sort.scala 23 additions, 5 deletions.../src/main/scala/org/apache/spark/sql/execution/sort.scala
Loading
Please register or sign in to comment