-
- Downloads
[SPARK-12559][SPARK SUBMIT] fix --packages for stand-alone cluster mode
Fixes --packages flag for the stand-alone case in cluster mode. Adds to the driver classpath the jars that are resolved via ivy along with any other jars passed to `spark.jars`. Jars not resolved by ivy are downloaded explicitly to a tmp folder on the driver node. Similar code is available in SparkSubmit so we refactored part of it to use it at the DriverWrapper class which is responsible for launching driver in standalone cluster mode. Note: In stand-alone mode `spark.jars` contains the user jar so it can be fetched later on at the executor side. Manually by submitting a driver in cluster mode within a standalone cluster and checking if dependencies were resolved at the driver side. Author: Stavros Kontopoulos <st.kontopoulos@gmail.com> Closes #18630 from skonto/fix_packages_stand_alone_cluster.
Showing
- core/src/main/scala/org/apache/spark/deploy/DependencyUtils.scala 99 additions, 0 deletions.../main/scala/org/apache/spark/deploy/DependencyUtils.scala
- core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala 15 additions, 37 deletions.../src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
- core/src/main/scala/org/apache/spark/deploy/worker/DriverWrapper.scala 23 additions, 0 deletions.../scala/org/apache/spark/deploy/worker/DriverWrapper.scala
Loading
Please register or sign in to comment