-
- Downloads
[SPARK-3772] Allow `ipython` to be used by Pyspark workers; IPython support improvements:
This pull request addresses a few issues related to PySpark's IPython support: - Fix the remaining uses of the '-u' flag, which IPython doesn't support (see SPARK-3772). - Change PYSPARK_PYTHON_OPTS to PYSPARK_DRIVER_PYTHON_OPTS, so that the old name is reserved in case we ever want to allow the worker Python options to be customized (this variable was introduced in #2554 and hasn't landed in a release yet, so this doesn't break any compatibility). - Introduce a PYSPARK_DRIVER_PYTHON option that allows the driver to use `ipython` while the workers use a different Python version. - Attempt to use Python 2.7 by default if PYSPARK_PYTHON is not specified. - Retain the old semantics for IPYTHON=1 and IPYTHON_OPTS (to avoid breaking existing example programs). There are more details in a block comment in `bin/pyspark`. Author: Josh Rosen <joshrosen@apache.org> Closes #2651 from JoshRosen/SPARK-3772 and squashes the following commits: 7b8eb86 [Josh Rosen] More changes to PySpark python executable configuration: c4f5778 [Josh Rosen] [SPARK-3772] Allow ipython to be used by Pyspark workers; IPython fixes:
Showing
- bin/pyspark 38 additions, 13 deletionsbin/pyspark
- core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala 6 additions, 2 deletions...ala/org/apache/spark/api/python/PythonWorkerFactory.scala
- core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala 3 additions, 1 deletion...src/main/scala/org/apache/spark/deploy/PythonRunner.scala
- docs/programming-guide.md 4 additions, 4 deletionsdocs/programming-guide.md
Please register or sign in to comment