sbin/spark-config.sh · c76baff0cc4775c2191d075cc9a8176e4915fec8 · cs525-sp18-g07 / spark

8 years ago

[SPARK-17336][PYSPARK] Fix appending multiple times to PYTHONPATH from spark-config.sh · c76baff0

Bryan Cutler authored 8 years ago

## What changes were proposed in this pull request?
During startup of Spark standalone, the script file spark-config.sh appends to the PYTHONPATH and can be sourced many times, causing duplicates in the path.  This change adds a env flag that is set when the PYTHONPATH is appended so it will happen only one time.

## How was this patch tested?
Manually started standalone master/worker and verified PYTHONPATH has no duplicate entries.

Author: Bryan Cutler <cutlerb@gmail.com>

Closes #15028 from BryanCutler/fix-duplicate-pythonpath-SPARK-17336.

c76baff0

History

[SPARK-17336][PYSPARK] Fix appending multiple times to PYTHONPATH from spark-config.sh

Bryan Cutler authored 8 years ago

## What changes were proposed in this pull request?
During startup of Spark standalone, the script file spark-config.sh appends to the PYTHONPATH and can be sourced many times, causing duplicates in the path.  This change adds a env flag that is set when the PYTHONPATH is appended so it will happen only one time.

## How was this patch tested?
Manually started standalone master/worker and verified PYTHONPATH has no duplicate entries.

Author: Bryan Cutler <cutlerb@gmail.com>

Closes #15028 from BryanCutler/fix-duplicate-pythonpath-SPARK-17336.

spark-config.sh 1.40 KiB