-
- Downloads
[SPARK-14437][CORE] Use the address that NettyBlockTransferService listens to create BlockManagerId
## What changes were proposed in this pull request? Here is why SPARK-14437 happens: BlockManagerId is created using NettyBlockTransferService.hostName which comes from `customHostname`. And `Executor` will set `customHostname` to the hostname which is detected by the driver. However, the driver may not be able to detect the correct address in some complicated network (Netty's Channel.remoteAddress doesn't always return a connectable address). In such case, `BlockManagerId` will be created using a wrong hostname. To fix this issue, this PR uses `hostname` provided by `SparkEnv.create` to create `NettyBlockTransferService` and set `NettyBlockTransferService.hostname` to this one directly. A bonus of this approach is NettyBlockTransferService won't bound to `0.0.0.0` which is much safer. ## How was this patch tested? Manually checked the bound address using local-cluster. Author: Shixiong Zhu <shixiong@databricks.com> Closes #12240 from zsxwing/SPARK-14437.
Showing
- core/src/main/scala/org/apache/spark/SparkEnv.scala 2 additions, 1 deletioncore/src/main/scala/org/apache/spark/SparkEnv.scala
- core/src/main/scala/org/apache/spark/network/netty/NettyBlockTransferService.scala 7 additions, 5 deletions...pache/spark/network/netty/NettyBlockTransferService.scala
- core/src/test/scala/org/apache/spark/network/netty/NettyBlockTransferSecuritySuite.scala 2 additions, 2 deletions...spark/network/netty/NettyBlockTransferSecuritySuite.scala
- core/src/test/scala/org/apache/spark/network/netty/NettyBlockTransferServiceSuite.scala 1 addition, 1 deletion.../spark/network/netty/NettyBlockTransferServiceSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerReplicationSuite.scala 1 addition, 1 deletion...g/apache/spark/storage/BlockManagerReplicationSuite.scala
- core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala 3 additions, 3 deletions...st/scala/org/apache/spark/storage/BlockManagerSuite.scala
- project/MimaExcludes.scala 3 additions, 0 deletionsproject/MimaExcludes.scala
- streaming/src/test/scala/org/apache/spark/streaming/ReceivedBlockHandlerSuite.scala 1 addition, 1 deletion...rg/apache/spark/streaming/ReceivedBlockHandlerSuite.scala
Loading
Please register or sign in to comment