-
- Downloads
[SPARK-20529][CORE] Allow worker and master work with a proxy server
## What changes were proposed in this pull request? In the current codes, when worker connects to master, master will send its address to the worker. Then worker will save this address and use it to reconnect in case of failure. However, sometimes, this address is not correct. If there is a proxy between master and worker, the address master sent is not the address of proxy. In this PR, the master address used by the worker will be sent to the master, then master just replies this address back, worker will use this address to reconnect in case of failure. In other words, the worker will use the config master address set in the worker side if possible rather than the master address set in the master side. There is still one potential issue though. When a master is restarted or takes over leadership, the work will use the address sent from the master to connect. If there is still a proxy between master and worker, the address may be wrong. However, there is no way to figure it out just in the worker. ## How was this patch tested? The new added unit test. Author: Shixiong Zhu <shixiong@databricks.com> Closes #17821 from zsxwing/SPARK-20529.
Showing
- core/src/main/scala/org/apache/spark/deploy/DeployMessage.scala 23 additions, 4 deletions...rc/main/scala/org/apache/spark/deploy/DeployMessage.scala
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 3 additions, 2 deletions...rc/main/scala/org/apache/spark/deploy/master/Master.scala
- core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala 45 additions, 8 deletions...rc/main/scala/org/apache/spark/deploy/worker/Worker.scala
- core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala 43 additions, 3 deletions...st/scala/org/apache/spark/deploy/master/MasterSuite.scala
Loading
Please register or sign in to comment