-
- Downloads
[SPARK-21321][SPARK CORE] Spark very verbose on shutdown
## What changes were proposed in this pull request? The current code is very verbose on shutdown. The changes I propose is to change the log level when the driver is shutting down and the RPC connections are closed (RpcEnvStoppedException). ## How was this patch tested? Tested with word count(deploy-mode = cluster, master = yarn, num-executors = 4) with 300GB of data. Author: John Lee <jlee2@yahoo-inc.com> Closes #18547 from yoonlee95/SPARK-21321.
Showing
- core/src/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala 5 additions, 2 deletions...rc/main/scala/org/apache/spark/rpc/netty/Dispatcher.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala 6 additions, 1 deletioncore/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala
- core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala 5 additions, 2 deletions...c/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala
- core/src/main/scala/org/apache/spark/rpc/netty/Outbox.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/rpc/netty/Outbox.scala
- core/src/main/scala/org/apache/spark/scheduler/LiveListenerBus.scala 1 addition, 1 deletion...in/scala/org/apache/spark/scheduler/LiveListenerBus.scala
- resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala 9 additions, 2 deletions...apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
Loading
Please register or sign in to comment