-
- Downloads
[SPARK-4737] Task set manager properly handles serialization errors
Dealing with [SPARK-4737], the handling of serialization errors should not be the DAGScheduler's responsibility. The task set manager now catches the error and aborts the stage. If the TaskSetManager throws a TaskNotSerializableException, the TaskSchedulerImpl will return an empty list of task descriptions, because no tasks were started. The scheduler should abort the stage gracefully. Note that I'm not too familiar with this part of the codebase and its place in the overall architecture of the Spark stack. If implementing it this way will have any averse side effects please voice that loudly. Author: mcheah <mcheah@palantir.com> Closes #3638 from mccheah/task-set-manager-properly-handle-ser-err and squashes the following commits: 1545984 [mcheah] Some more style fixes from Andrew Or. 5267929 [mcheah] Fixing style suggestions from Andrew Or. dfa145b [mcheah] Fixing style from Josh Rosen's feedback b2a430d [mcheah] Not returning empty seq when a task set cannot be serialized. 94844d7 [mcheah] Fixing compilation error, one brace too many 5f486f4 [mcheah] Adding license header for fake task class bf5e706 [mcheah] Fixing indentation. 097e7a2 [mcheah] [SPARK-4737] Catching task serialization exception in TaskSetManager
Showing
- core/src/main/scala/org/apache/spark/TaskNotSerializableException.scala 25 additions, 0 deletions...scala/org/apache/spark/TaskNotSerializableException.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 0 additions, 20 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 37 additions, 17 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 14 additions, 4 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/test/scala/org/apache/spark/SharedSparkContext.scala 1 addition, 1 deletion.../src/test/scala/org/apache/spark/SharedSparkContext.scala
- core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala 21 additions, 0 deletionscore/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/NotSerializableFakeTask.scala 40 additions, 0 deletions.../org/apache/spark/scheduler/NotSerializableFakeTask.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala 30 additions, 0 deletions...a/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 14 additions, 0 deletions...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
Loading
Please register or sign in to comment