-
- Downloads
[SPARK-7655][Core][SQL] Remove...
[SPARK-7655][Core][SQL] Remove 'scala.concurrent.ExecutionContext.Implicits.global' in 'ask' and 'BroadcastHashJoin' Because both `AkkaRpcEndpointRef.ask` and `BroadcastHashJoin` uses `scala.concurrent.ExecutionContext.Implicits.global`. However, because the tasks in `BroadcastHashJoin` are usually long-running tasks, which will occupy all threads in `global`. Then `ask` cannot get a chance to process the replies. For `ask`, actually the tasks are very simple, so we can use `MoreExecutors.sameThreadExecutor()`. For `BroadcastHashJoin`, it's better to use `ThreadUtils.newDaemonCachedThreadPool`. Author: zsxwing <zsxwing@gmail.com> Closes #6200 from zsxwing/SPARK-7655-2 and squashes the following commits: cfdc605 [zsxwing] Remove redundant imort and minor doc fix cf83153 [zsxwing] Add "sameThread" and "newDaemonCachedThreadPool with maxThreadNumber" to ThreadUtils 08ad0ee [zsxwing] Remove 'scala.concurrent.ExecutionContext.Implicits.global' in 'ask' and 'BroadcastHashJoin'
Showing
- core/src/main/scala/org/apache/spark/rpc/akka/AkkaRpcEnv.scala 5 additions, 3 deletions...src/main/scala/org/apache/spark/rpc/akka/AkkaRpcEnv.scala
- core/src/main/scala/org/apache/spark/util/ThreadUtils.scala 23 additions, 1 deletioncore/src/main/scala/org/apache/spark/util/ThreadUtils.scala
- core/src/test/scala/org/apache/spark/util/ThreadUtilsSuite.scala 12 additions, 0 deletions...c/test/scala/org/apache/spark/util/ThreadUtilsSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoin.scala 8 additions, 2 deletions.../apache/spark/sql/execution/joins/BroadcastHashJoin.scala
Loading
Please register or sign in to comment