Skip to content
Snippets Groups Projects
  • Michael Nazario's avatar
    1c5b1982
    [SPARK-7899] [PYSPARK] Fix Python 3 pyspark/sql/types module conflict · 1c5b1982
    Michael Nazario authored
    This PR makes the types module in `pyspark/sql/types` work with pylint static analysis by removing the dynamic naming of the `pyspark/sql/_types` module to `pyspark/sql/types`.
    
    Tests are now loaded using `$PYSPARK_DRIVER_PYTHON -m module` rather than `$PYSPARK_DRIVER_PYTHON module.py`. The old method adds the location of `module.py` to `sys.path`, so this change prevents accidental use of relative paths in Python.
    
    Author: Michael Nazario <mnazario@palantir.com>
    
    Closes #6439 from mnazario/feature/SPARK-7899 and squashes the following commits:
    
    366ef30 [Michael Nazario] Remove hack on random.py
    bb8b04d [Michael Nazario] Make doctests consistent with other tests
    6ee4f75 [Michael Nazario] Change test scripts to use "-m"
    673528f [Michael Nazario] Move _types back to types
    1c5b1982
    History
    [SPARK-7899] [PYSPARK] Fix Python 3 pyspark/sql/types module conflict
    Michael Nazario authored
    This PR makes the types module in `pyspark/sql/types` work with pylint static analysis by removing the dynamic naming of the `pyspark/sql/_types` module to `pyspark/sql/types`.
    
    Tests are now loaded using `$PYSPARK_DRIVER_PYTHON -m module` rather than `$PYSPARK_DRIVER_PYTHON module.py`. The old method adds the location of `module.py` to `sys.path`, so this change prevents accidental use of relative paths in Python.
    
    Author: Michael Nazario <mnazario@palantir.com>
    
    Closes #6439 from mnazario/feature/SPARK-7899 and squashes the following commits:
    
    366ef30 [Michael Nazario] Remove hack on random.py
    bb8b04d [Michael Nazario] Make doctests consistent with other tests
    6ee4f75 [Michael Nazario] Change test scripts to use "-m"
    673528f [Michael Nazario] Move _types back to types
pyspark 3.55 KiB