Skip to content
Snippets Groups Projects
Commit 1b0c8e61 authored by huangzhaowei's avatar huangzhaowei Committed by Andrew Or
Browse files

[SPARK-8687] [YARN] Fix bug: Executor can't fetch the new set configuration in yarn-client

Spark initi the properties CoarseGrainedSchedulerBackend.start
```scala
    // TODO (prashant) send conf instead of properties
    driverEndpoint = rpcEnv.setupEndpoint(
      CoarseGrainedSchedulerBackend.ENDPOINT_NAME, new DriverEndpoint(rpcEnv, properties))
```
Then the yarn logic will set some configuration but not update in this `properties`.
So `Executor` won't gain the `properties`.

[Jira](https://issues.apache.org/jira/browse/SPARK-8687)

Author: huangzhaowei <carlmartinmax@gmail.com>

Closes #7066 from SaintBacchus/SPARK-8687 and squashes the following commits:

1de4f48 [huangzhaowei] Ensure all necessary properties have already been set before startup ExecutorLaucher
parent 3697232b
No related branches found
No related tags found
No related merge requests found
......@@ -41,7 +41,6 @@ private[spark] class YarnClientSchedulerBackend(
* This waits until the application is running.
*/
override def start() {
super.start()
val driverHost = conf.get("spark.driver.host")
val driverPort = conf.get("spark.driver.port")
val hostport = driverHost + ":" + driverPort
......@@ -56,6 +55,12 @@ private[spark] class YarnClientSchedulerBackend(
totalExpectedExecutors = args.numExecutors
client = new Client(args, conf)
appId = client.submitApplication()
// SPARK-8687: Ensure all necessary properties have already been set before
// we initialize our driver scheduler backend, which serves these properties
// to the executors
super.start()
waitForApplication()
monitorThread = asyncMonitorApplication()
monitorThread.start()
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment