-
- Downloads
[SPARK-3466] Limit size of results that a driver collects for each action
Right now, operations like collect() and take() can crash the driver with an OOM if they bring back too many data. This PR will introduce spark.driver.maxResultSize, after setting it, the driver will abort a job if its result is bigger than it. By default, it's 1g (for backward compatibility for most the cases). In local mode, the driver and executor share the same JVM, the default setting can not protect JVM from OOM. cc mateiz Author: Davies Liu <davies@databricks.com> Closes #3003 from davies/collect and squashes the following commits: 248ed5e [Davies Liu] fix compile 272522e [Davies Liu] address comments 2c35773 [Davies Liu] add sizes in message of abort() 5d62303 [Davies Liu] address comments bc3c077 [Davies Liu] Merge branch 'master' of github.com:apache/spark into collect 11f97c5 [Davies Liu] address comments 47b144f [Davies Liu] check the size of result before send and fetch 3d81af2 [Davies Liu] address comments ca8267d [Davies Liu] limit the size of data by collect
Showing
- core/src/main/scala/org/apache/spark/executor/Executor.scala 15 additions, 10 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala 2 additions, 2 deletions...rc/main/scala/org/apache/spark/scheduler/TaskResult.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskResultGetter.scala 15 additions, 5 deletions...n/scala/org/apache/spark/scheduler/TaskResultGetter.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 29 additions, 4 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 5 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskResultGetterSuite.scala 1 addition, 1 deletion...la/org/apache/spark/scheduler/TaskResultGetterSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 25 additions, 0 deletions...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
- docs/configuration.md 12 additions, 0 deletionsdocs/configuration.md
Loading
Please register or sign in to comment