Skip to content
Snippets Groups Projects
  • Holden Karau's avatar
    12c937ed
    [SPARK-20627][PYSPARK] Drop the hadoop distirbution name from the Python version · 12c937ed
    Holden Karau authored
    ## What changes were proposed in this pull request?
    
    Drop the hadoop distirbution name from the Python version (PEP440 - https://www.python.org/dev/peps/pep-0440/
    
    ). We've been using the local version string to disambiguate between different hadoop versions packaged with PySpark, but PEP0440 states that local versions should not be used when publishing up-stream. Since we no longer make PySpark pip packages for different hadoop versions, we can simply drop the hadoop information. If at a later point we need to start publishing different hadoop versions we can look at make different packages or similar.
    
    ## How was this patch tested?
    
    Ran `make-distribution` locally
    
    Author: Holden Karau <holden@us.ibm.com>
    
    Closes #17885 from holdenk/SPARK-20627-remove-pip-local-version-string.
    
    (cherry picked from commit 1b85bcd9)
    Signed-off-by: default avatarHolden Karau <holden@us.ibm.com>
    12c937ed
    History
    [SPARK-20627][PYSPARK] Drop the hadoop distirbution name from the Python version
    Holden Karau authored
    ## What changes were proposed in this pull request?
    
    Drop the hadoop distirbution name from the Python version (PEP440 - https://www.python.org/dev/peps/pep-0440/
    
    ). We've been using the local version string to disambiguate between different hadoop versions packaged with PySpark, but PEP0440 states that local versions should not be used when publishing up-stream. Since we no longer make PySpark pip packages for different hadoop versions, we can simply drop the hadoop information. If at a later point we need to start publishing different hadoop versions we can look at make different packages or similar.
    
    ## How was this patch tested?
    
    Ran `make-distribution` locally
    
    Author: Holden Karau <holden@us.ibm.com>
    
    Closes #17885 from holdenk/SPARK-20627-remove-pip-local-version-string.
    
    (cherry picked from commit 1b85bcd9)
    Signed-off-by: default avatarHolden Karau <holden@us.ibm.com>
release-build.sh 14.47 KiB