-
- Downloads
[SPARK-13845][CORE] Using onBlockUpdated to replace onTaskEnd avioding driver OOM
## What changes were proposed in this pull request? We have a streaming job using `FlumePollInputStream` always driver OOM after few days, here is some driver heap dump before OOM ``` num #instances #bytes class name ---------------------------------------------- 1: 13845916 553836640 org.apache.spark.storage.BlockStatus 2: 14020324 336487776 org.apache.spark.storage.StreamBlockId 3: 13883881 333213144 scala.collection.mutable.DefaultEntry 4: 8907 89043952 [Lscala.collection.mutable.HashEntry; 5: 62360 65107352 [B 6: 163368 24453904 [Ljava.lang.Object; 7: 293651 20342664 [C ... ``` `BlockStatus` and `StreamBlockId` keep on growing, and the driver OOM in the end. After investigated, i found the `executorIdToStorageStatus` in `StorageStatusListener` seems never remove the blocks from `StorageStatus`. In order to fix the issue, i try to use `onBlockUpdated` replace `onTaskEnd ` , so we can update the block informations(add blocks, drop the block from memory to disk and delete the blocks) in time. ## How was this patch tested? Existing unit tests and manual tests Author: jeanlyn <jeanlyn92@gmail.com> Closes #11779 from jeanlyn/fix_driver_oom.
Showing
- core/src/main/scala/org/apache/spark/storage/StorageStatusListener.scala 10 additions, 11 deletions...cala/org/apache/spark/storage/StorageStatusListener.scala
- core/src/main/scala/org/apache/spark/ui/storage/StorageTab.scala 10 additions, 11 deletions...c/main/scala/org/apache/spark/ui/storage/StorageTab.scala
- core/src/test/resources/HistoryServerExpectations/executor_list_json_expectation.json 2 additions, 2 deletions...oryServerExpectations/executor_list_json_expectation.json
- core/src/test/resources/HistoryServerExpectations/rdd_list_storage_json_expectation.json 1 addition, 9 deletions...ServerExpectations/rdd_list_storage_json_expectation.json
- core/src/test/scala/org/apache/spark/deploy/history/HistoryServerSuite.scala 3 additions, 2 deletions.../org/apache/spark/deploy/history/HistoryServerSuite.scala
- core/src/test/scala/org/apache/spark/storage/StorageStatusListenerSuite.scala 38 additions, 29 deletions...org/apache/spark/storage/StorageStatusListenerSuite.scala
- core/src/test/scala/org/apache/spark/ui/storage/StorageTabSuite.scala 27 additions, 31 deletions...t/scala/org/apache/spark/ui/storage/StorageTabSuite.scala
Loading
Please register or sign in to comment