-
- Downloads
[SPARK-13747][CORE] Fix potential ThreadLocal leaks in RPC when using ForkJoinPool
## What changes were proposed in this pull request? Some places in SQL may call `RpcEndpointRef.askWithRetry` (e.g., ParquetFileFormat.buildReader -> SparkContext.broadcast -> ... -> BlockManagerMaster.updateBlockInfo -> RpcEndpointRef.askWithRetry), which will finally call `Await.result`. It may cause `java.lang.IllegalArgumentException: spark.sql.execution.id is already set` when running in Scala ForkJoinPool. This PR includes the following changes to fix this issue: - Remove `ThreadUtils.awaitResult` - Rename `ThreadUtils. awaitResultInForkJoinSafely` to `ThreadUtils.awaitResult` - Replace `Await.result` in RpcTimeout with `ThreadUtils.awaitResult`. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes #16230 from zsxwing/fix-SPARK-13747.
Showing
- core/src/main/scala/org/apache/spark/rpc/RpcTimeout.scala 3 additions, 9 deletionscore/src/main/scala/org/apache/spark/rpc/RpcTimeout.scala
- core/src/main/scala/org/apache/spark/util/ThreadUtils.scala 16 additions, 25 deletionscore/src/main/scala/org/apache/spark/util/ThreadUtils.scala
- core/src/test/scala/org/apache/spark/rdd/AsyncRDDActionsSuite.scala 1 addition, 2 deletions...est/scala/org/apache/spark/rdd/AsyncRDDActionsSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala 1 addition, 2 deletions...apache/spark/scheduler/OutputCommitCoordinatorSuite.scala
- scalastyle-config.xml 0 additions, 1 deletionscalastyle-config.xml
- sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala 1 addition, 1 deletion...g/apache/spark/sql/execution/basicPhysicalOperators.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala 1 addition, 2 deletions.../spark/sql/execution/exchange/BroadcastExchangeExec.scala
Loading
Please register or sign in to comment