Skip to content
Snippets Groups Projects
Commit 640afa49 authored by Li Yichao's avatar Li Yichao Committed by Marcelo Vanzin
Browse files

[SPARK-20365][YARN] Remove local scheme when add path to ClassPath.

In Spark on YARN, when configuring "spark.yarn.jars" with local jars (jars started with "local" scheme), we will get inaccurate classpath for AM and containers. This is because we don't remove "local" scheme when concatenating classpath. It is OK to run because classpath is separated with ":" and java treat "local" as a separate jar. But we could improve it to remove the scheme.

Updated `ClientSuite` to check "local" is not in the classpath.

cc jerryshao

Author: Li Yichao <lyc@zhihu.com>
Author: Li Yichao <liyichao.good@gmail.com>

Closes #18129 from liyichao/SPARK-20365.
parent f7cf2096
No related branches found
No related tags found
No related merge requests found
...@@ -1275,7 +1275,8 @@ private object Client extends Logging { ...@@ -1275,7 +1275,8 @@ private object Client extends Logging {
if (sparkConf.get(SPARK_ARCHIVE).isEmpty) { if (sparkConf.get(SPARK_ARCHIVE).isEmpty) {
sparkConf.get(SPARK_JARS).foreach { jars => sparkConf.get(SPARK_JARS).foreach { jars =>
jars.filter(isLocalUri).foreach { jar => jars.filter(isLocalUri).foreach { jar =>
addClasspathEntry(getClusterPath(sparkConf, jar), env) val uri = new URI(jar)
addClasspathEntry(getClusterPath(sparkConf, uri.getPath()), env)
} }
} }
} }
......
...@@ -122,6 +122,7 @@ class ClientSuite extends SparkFunSuite with Matchers with BeforeAndAfterAll ...@@ -122,6 +122,7 @@ class ClientSuite extends SparkFunSuite with Matchers with BeforeAndAfterAll
cp should not contain (uri.getPath()) cp should not contain (uri.getPath())
} }
}) })
cp should not contain ("local")
cp should contain(PWD) cp should contain(PWD)
cp should contain (s"$PWD${Path.SEPARATOR}${LOCALIZED_CONF_DIR}") cp should contain (s"$PWD${Path.SEPARATOR}${LOCALIZED_CONF_DIR}")
cp should not contain (APP_JAR) cp should not contain (APP_JAR)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment