Skip to content
Snippets Groups Projects
  • Rob O'Dwyer's avatar
    f38fab97
    SPARK-3265 Allow using custom ipython executable with pyspark · f38fab97
    Rob O'Dwyer authored
    Although you can make pyspark use ipython with `IPYTHON=1`, and also change the python executable with `PYSPARK_PYTHON=...`, you can't use both at the same time because it hardcodes the default ipython script.
    
    This makes it use the `PYSPARK_PYTHON` variable if present and fall back to default python, similarly to how the default python executable is handled.
    
    So you can use a custom ipython like so:
    `PYSPARK_PYTHON=./anaconda/bin/ipython IPYTHON_OPTS="notebook" pyspark`
    
    Author: Rob O'Dwyer <odwyerrob@gmail.com>
    
    Closes #2167 from robbles/patch-1 and squashes the following commits:
    
    d98e8a9 [Rob O'Dwyer] Allow using custom ipython executable with pyspark
    f38fab97
    History
    SPARK-3265 Allow using custom ipython executable with pyspark
    Rob O'Dwyer authored
    Although you can make pyspark use ipython with `IPYTHON=1`, and also change the python executable with `PYSPARK_PYTHON=...`, you can't use both at the same time because it hardcodes the default ipython script.
    
    This makes it use the `PYSPARK_PYTHON` variable if present and fall back to default python, similarly to how the default python executable is handled.
    
    So you can use a custom ipython like so:
    `PYSPARK_PYTHON=./anaconda/bin/ipython IPYTHON_OPTS="notebook" pyspark`
    
    Author: Rob O'Dwyer <odwyerrob@gmail.com>
    
    Closes #2167 from robbles/patch-1 and squashes the following commits:
    
    d98e8a9 [Rob O'Dwyer] Allow using custom ipython executable with pyspark