Skip to content
Snippets Groups Projects
  • Shixiong Zhu's avatar
    fb3081d3
    [SPARK-13747][CORE] Fix potential ThreadLocal leaks in RPC when using ForkJoinPool · fb3081d3
    Shixiong Zhu authored
    ## What changes were proposed in this pull request?
    
    Some places in SQL may call `RpcEndpointRef.askWithRetry` (e.g., ParquetFileFormat.buildReader -> SparkContext.broadcast -> ... -> BlockManagerMaster.updateBlockInfo -> RpcEndpointRef.askWithRetry), which will finally call `Await.result`. It may cause `java.lang.IllegalArgumentException: spark.sql.execution.id is already set` when running in Scala ForkJoinPool.
    
    This PR includes the following changes to fix this issue:
    
    - Remove `ThreadUtils.awaitResult`
    - Rename `ThreadUtils. awaitResultInForkJoinSafely` to `ThreadUtils.awaitResult`
    - Replace `Await.result` in RpcTimeout with `ThreadUtils.awaitResult`.
    
    ## How was this patch tested?
    
    Jenkins
    
    Author: Shixiong Zhu <shixiong@databricks.com>
    
    Closes #16230 from zsxwing/fix-SPARK-13747.
    fb3081d3
    History
    [SPARK-13747][CORE] Fix potential ThreadLocal leaks in RPC when using ForkJoinPool
    Shixiong Zhu authored
    ## What changes were proposed in this pull request?
    
    Some places in SQL may call `RpcEndpointRef.askWithRetry` (e.g., ParquetFileFormat.buildReader -> SparkContext.broadcast -> ... -> BlockManagerMaster.updateBlockInfo -> RpcEndpointRef.askWithRetry), which will finally call `Await.result`. It may cause `java.lang.IllegalArgumentException: spark.sql.execution.id is already set` when running in Scala ForkJoinPool.
    
    This PR includes the following changes to fix this issue:
    
    - Remove `ThreadUtils.awaitResult`
    - Rename `ThreadUtils. awaitResultInForkJoinSafely` to `ThreadUtils.awaitResult`
    - Replace `Await.result` in RpcTimeout with `ThreadUtils.awaitResult`.
    
    ## How was this patch tested?
    
    Jenkins
    
    Author: Shixiong Zhu <shixiong@databricks.com>
    
    Closes #16230 from zsxwing/fix-SPARK-13747.