-
- Downloads
[SPARK-14676] Wrap and re-throw Await.result exceptions in order to capture full stacktrace
When `Await.result` throws an exception which originated from a different thread, the resulting stacktrace doesn't include the path leading to the `Await.result` call itself, making it difficult to identify the impact of these exceptions. For example, I've seen cases where broadcast cleaning errors propagate to the main thread and crash it but the resulting stacktrace doesn't include any of the main thread's code, making it difficult to pinpoint which exception crashed that thread. This patch addresses this issue by explicitly catching, wrapping, and re-throwing exceptions that are thrown by `Await.result`. I tested this manually using https://github.com/JoshRosen/spark/commit/16b31c825197ee31a50214c6ba3c1df08148f403, a patch which reproduces an issue where an RPC exception which occurs while unpersisting RDDs manages to crash the main thread without any useful stacktrace, and verified that informative, full stacktraces were generated after applying the fix in this PR. /cc rxin nongli yhuai anabranch Author: Josh Rosen <joshrosen@databricks.com> Closes #12433 from JoshRosen/wrap-and-rethrow-await-exceptions.
Showing
- core/src/main/scala/org/apache/spark/FutureAction.scala 5 additions, 2 deletionscore/src/main/scala/org/apache/spark/FutureAction.scala
- core/src/main/scala/org/apache/spark/deploy/FaultToleranceTest.scala 5 additions, 5 deletions...in/scala/org/apache/spark/deploy/FaultToleranceTest.scala
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 2 additions, 2 deletions...rc/main/scala/org/apache/spark/deploy/master/Master.scala
- core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala 6 additions, 1 deletion...a/org/apache/spark/deploy/rest/RestSubmissionClient.scala
- core/src/main/scala/org/apache/spark/network/BlockTransferService.scala 4 additions, 4 deletions...scala/org/apache/spark/network/BlockTransferService.scala
- core/src/main/scala/org/apache/spark/rpc/RpcTimeout.scala 18 additions, 7 deletionscore/src/main/scala/org/apache/spark/rpc/RpcTimeout.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 12 additions, 2 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/util/ThreadUtils.scala 21 additions, 1 deletioncore/src/main/scala/org/apache/spark/util/ThreadUtils.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 2 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/FutureActionSuite.scala 4 additions, 3 deletionscore/src/test/scala/org/apache/spark/FutureActionSuite.scala
- core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala 3 additions, 4 deletions.../test/scala/org/apache/spark/HeartbeatReceiverSuite.scala
- core/src/test/scala/org/apache/spark/JobCancellationSuite.scala 7 additions, 7 deletions...rc/test/scala/org/apache/spark/JobCancellationSuite.scala
- core/src/test/scala/org/apache/spark/memory/MemoryManagerSuite.scala 22 additions, 21 deletions...st/scala/org/apache/spark/memory/MemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/rdd/AsyncRDDActionsSuite.scala 7 additions, 5 deletions...est/scala/org/apache/spark/rdd/AsyncRDDActionsSuite.scala
- core/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala 14 additions, 11 deletionscore/src/test/scala/org/apache/spark/rpc/RpcEnvSuite.scala
- core/src/test/scala/org/apache/spark/rpc/netty/NettyRpcEnvSuite.scala 4 additions, 3 deletions...t/scala/org/apache/spark/rpc/netty/NettyRpcEnvSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala 4 additions, 4 deletions...apache/spark/scheduler/OutputCommitCoordinatorSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockInfoManagerSuite.scala 12 additions, 10 deletions...cala/org/apache/spark/storage/BlockInfoManagerSuite.scala
- core/src/test/scala/org/apache/spark/util/ThreadUtilsSuite.scala 2 additions, 2 deletions...c/test/scala/org/apache/spark/util/ThreadUtilsSuite.scala
- scalastyle-config.xml 11 additions, 0 deletionsscalastyle-config.xml
Loading
Please register or sign in to comment