Skip to content
Snippets Groups Projects
Commit 6343f665 authored by MechCoder's avatar MechCoder Committed by Sean Owen
Browse files

[SPARK-16399][PYSPARK] Force PYSPARK_PYTHON to python

## What changes were proposed in this pull request?

I would like to change

```bash
if hash python2.7 2>/dev/null; then
  # Attempt to use Python 2.7, if installed:
  DEFAULT_PYTHON="python2.7"
else
  DEFAULT_PYTHON="python"
fi
```

to just ```DEFAULT_PYTHON="python"```

I'm not sure if it is a great assumption that python2.7 is used by default, when python points to something else.

## How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Author: MechCoder <mks542@nyu.edu>

Closes #14016 from MechCoder/followup.
parent 4c6f00d0
No related branches found
No related tags found
No related merge requests found
...@@ -30,14 +30,6 @@ export _SPARK_CMD_USAGE="Usage: ./bin/pyspark [options]" ...@@ -30,14 +30,6 @@ export _SPARK_CMD_USAGE="Usage: ./bin/pyspark [options]"
# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook'). This supports full customization of the IPython # (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook'). This supports full customization of the IPython
# and executor Python executables. # and executor Python executables.
# Determine the Python executable to use if PYSPARK_PYTHON or PYSPARK_DRIVER_PYTHON isn't set:
if hash python2.7 2>/dev/null; then
# Attempt to use Python 2.7, if installed:
DEFAULT_PYTHON="python2.7"
else
DEFAULT_PYTHON="python"
fi
# Fail noisily if removed options are set # Fail noisily if removed options are set
if [[ -n "$IPYTHON" || -n "$IPYTHON_OPTS" ]]; then if [[ -n "$IPYTHON" || -n "$IPYTHON_OPTS" ]]; then
echo "Error in pyspark startup:" echo "Error in pyspark startup:"
...@@ -47,10 +39,10 @@ fi ...@@ -47,10 +39,10 @@ fi
# Default to standard python interpreter unless told otherwise # Default to standard python interpreter unless told otherwise
if [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then if [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then
PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"$DEFAULT_PYTHON"}" PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"python"}"
fi fi
WORKS_WITH_IPYTHON=$($DEFAULT_PYTHON -c 'import sys; print(sys.version_info >= (2, 7, 0))') WORKS_WITH_IPYTHON=$(python -c 'import sys; print(sys.version_info >= (2, 7, 0))')
# Determine the Python executable to use for the executors: # Determine the Python executable to use for the executors:
if [[ -z "$PYSPARK_PYTHON" ]]; then if [[ -z "$PYSPARK_PYTHON" ]]; then
...@@ -58,7 +50,7 @@ if [[ -z "$PYSPARK_PYTHON" ]]; then ...@@ -58,7 +50,7 @@ if [[ -z "$PYSPARK_PYTHON" ]]; then
echo "IPython requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON" 1>&2 echo "IPython requires Python 2.7+; please install python2.7 or set PYSPARK_PYTHON" 1>&2
exit 1 exit 1
else else
PYSPARK_PYTHON="$DEFAULT_PYTHON" PYSPARK_PYTHON=python
fi fi
fi fi
export PYSPARK_PYTHON export PYSPARK_PYTHON
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment