-
- Downloads
[SPARK-20365][YARN] Remove local scheme when add path to ClassPath.
In Spark on YARN, when configuring "spark.yarn.jars" with local jars (jars started with "local" scheme), we will get inaccurate classpath for AM and containers. This is because we don't remove "local" scheme when concatenating classpath. It is OK to run because classpath is separated with ":" and java treat "local" as a separate jar. But we could improve it to remove the scheme. Updated `ClientSuite` to check "local" is not in the classpath. cc jerryshao Author: Li Yichao <lyc@zhihu.com> Author: Li Yichao <liyichao.good@gmail.com> Closes #18129 from liyichao/SPARK-20365.
Showing
- resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala 2 additions, 1 deletion.../src/main/scala/org/apache/spark/deploy/yarn/Client.scala
- resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala 1 addition, 0 deletions...test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
Loading
Please register or sign in to comment