-
- Downloads
[SPARK-13081][PYSPARK][SPARK_SUBMIT] Allow set pythonExec of driver and executor through conf…
Before this PR, user have to export environment variable to specify the python of driver & executor which is not so convenient for users. This PR is trying to allow user to specify python through configuration "--pyspark-driver-python" & "--pyspark-executor-python" Manually test in local & yarn mode for pyspark-shell and pyspark batch mode. Author: Jeff Zhang <zjffdu@apache.org> Closes #13146 from zjffdu/SPARK-13081.
Showing
- core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala 11 additions, 3 deletions...src/main/scala/org/apache/spark/deploy/PythonRunner.scala
- core/src/main/scala/org/apache/spark/internal/config/package.scala 8 additions, 0 deletions...main/scala/org/apache/spark/internal/config/package.scala
- core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java 8 additions, 0 deletions...st/java/org/apache/spark/launcher/SparkLauncherSuite.java
- core/src/test/scala/org/apache/spark/SparkConfSuite.scala 2 additions, 0 deletionscore/src/test/scala/org/apache/spark/SparkConfSuite.scala
- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala 5 additions, 0 deletions...test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
- docs/configuration.md 19 additions, 2 deletionsdocs/configuration.md
- launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java 4 additions, 0 deletions...rc/main/java/org/apache/spark/launcher/SparkLauncher.java
- launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java 15 additions, 3 deletions.../org/apache/spark/launcher/SparkSubmitCommandBuilder.java
Loading
Please register or sign in to comment