-
- Downloads
[SPARK-21714][CORE][YARN] Avoiding re-uploading remote resources in yarn client mode
## What changes were proposed in this pull request? With SPARK-10643, Spark supports download resources from remote in client deploy mode. But the implementation overrides variables which representing added resources (like `args.jars`, `args.pyFiles`) to local path, And yarn client leverage this local path to re-upload resources to distributed cache. This is unnecessary to break the semantics of putting resources in a shared FS. So here proposed to fix it. ## How was this patch tested? This is manually verified with jars, pyFiles in local and remote storage, both in client and cluster mode. Author: jerryshao <sshao@hortonworks.com> Closes #18962 from jerryshao/SPARK-21714.
Showing
- core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala 41 additions, 23 deletions.../src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
- core/src/main/scala/org/apache/spark/internal/config/package.scala 1 addition, 1 deletion...main/scala/org/apache/spark/internal/config/package.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 15 additions, 10 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala 56 additions, 14 deletions...test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
- repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala 1 addition, 1 deletion...cala-2.11/src/main/scala/org/apache/spark/repl/Main.scala
Loading
Please register or sign in to comment