Skip to content
Snippets Groups Projects
  • hyukjinkwon's avatar
    46b21260
    [SPARK-19002][BUILD][PYTHON] Check pep8 against all Python scripts · 46b21260
    hyukjinkwon authored
    ## What changes were proposed in this pull request?
    
    This PR proposes to check pep8 against all other Python scripts and fix the errors as below:
    
    ```bash
    ./dev/create-release/generate-contributors.py
    ./dev/create-release/releaseutils.py
    ./dev/create-release/translate-contributors.py
    ./dev/lint-python
    ./python/docs/epytext.py
    ./examples/src/main/python/mllib/decision_tree_classification_example.py
    ./examples/src/main/python/mllib/decision_tree_regression_example.py
    ./examples/src/main/python/mllib/gradient_boosting_classification_example.py
    ./examples/src/main/python/mllib/gradient_boosting_regression_example.py
    ./examples/src/main/python/mllib/linear_regression_with_sgd_example.py
    ./examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py
    ./examples/src/main/python/mllib/naive_bayes_example.py
    ./examples/src/main/python/mllib/random_forest_classification_example.py
    ./examples/src/main/python/mllib/random_forest_regression_example.py
    ./examples/src/main/python/mllib/svm_with_sgd_example.py
    ./examples/src/main/python/streaming/network_wordjoinsentiments.py
    ./sql/hive/src/test/resources/data/scripts/cat.py
    ./sql/hive/src/test/resources/data/scripts/cat_error.py
    ./sql/hive/src/test/resources/data/scripts/doubleescapedtab.py
    ./sql/hive/src/test/resources/data/scripts/dumpdata_script.py
    ./sql/hive/src/test/resources/data/scripts/escapedcarriagereturn.py
    ./sql/hive/src/test/resources/data/scripts/escapednewline.py
    ./sql/hive/src/test/resources/data/scripts/escapedtab.py
    ./sql/hive/src/test/resources/data/scripts/input20_script.py
    ./sql/hive/src/test/resources/data/scripts/newline.py
    ```
    
    ## How was this patch tested?
    
    - `./python/docs/epytext.py`
    
      ```bash
      cd ./python/docs $$ make html
      ```
    
    - pep8 check (Python 2.7 / Python 3.3.6)
    
      ```
      ./dev/lint-python
      ```
    
    - `./dev/merge_spark_pr.py` (Python 2.7 only / Python 3.3.6 not working)
    
      ```bash
      python -m doctest -v ./dev/merge_spark_pr.py
      ```
    
    - `./dev/create-release/releaseutils.py` `./dev/create-release/generate-contributors.py` `./dev/create-release/translate-contributors.py` (Python 2.7 only / Python 3.3.6 not working)
    
      ```bash
      python generate-contributors.py
      python translate-contributors.py
      ```
    
    - Examples (Python 2.7 / Python 3.3.6)
    
      ```bash
      ./bin/spark-submit examples/src/main/python/mllib/decision_tree_classification_example.py
      ./bin/spark-submit examples/src/main/python/mllib/decision_tree_regression_example.py
      ./bin/spark-submit examples/src/main/python/mllib/gradient_boosting_classification_example.py
      ./bin/spark-submit examples/src/main/python/mllib/gradient_boosting_regression_example.p
      ./bin/spark-submit examples/src/main/python/mllib/random_forest_classification_example.py
      ./bin/spark-submit examples/src/main/python/mllib/random_forest_regression_example.py
      ```
    
    - Examples (Python 2.7 only / Python 3.3.6 not working)
      ```
      ./bin/spark-submit examples/src/main/python/mllib/linear_regression_with_sgd_example.py
      ./bin/spark-submit examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py
      ./bin/spark-submit examples/src/main/python/mllib/naive_bayes_example.py
      ./bin/spark-submit examples/src/main/python/mllib/svm_with_sgd_example.py
      ```
    
    - `sql/hive/src/test/resources/data/scripts/*.py` (Python 2.7 / Python 3.3.6 within suggested changes)
    
      Manually tested only changed ones.
    
    - `./dev/github_jira_sync.py` (Python 2.7 only / Python 3.3.6 not working)
    
      Manually tested this after disabling actually adding comments and links.
    
    And also via Jenkins tests.
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #16405 from HyukjinKwon/minor-pep8.
    [SPARK-19002][BUILD][PYTHON] Check pep8 against all Python scripts
    hyukjinkwon authored
    ## What changes were proposed in this pull request?
    
    This PR proposes to check pep8 against all other Python scripts and fix the errors as below:
    
    ```bash
    ./dev/create-release/generate-contributors.py
    ./dev/create-release/releaseutils.py
    ./dev/create-release/translate-contributors.py
    ./dev/lint-python
    ./python/docs/epytext.py
    ./examples/src/main/python/mllib/decision_tree_classification_example.py
    ./examples/src/main/python/mllib/decision_tree_regression_example.py
    ./examples/src/main/python/mllib/gradient_boosting_classification_example.py
    ./examples/src/main/python/mllib/gradient_boosting_regression_example.py
    ./examples/src/main/python/mllib/linear_regression_with_sgd_example.py
    ./examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py
    ./examples/src/main/python/mllib/naive_bayes_example.py
    ./examples/src/main/python/mllib/random_forest_classification_example.py
    ./examples/src/main/python/mllib/random_forest_regression_example.py
    ./examples/src/main/python/mllib/svm_with_sgd_example.py
    ./examples/src/main/python/streaming/network_wordjoinsentiments.py
    ./sql/hive/src/test/resources/data/scripts/cat.py
    ./sql/hive/src/test/resources/data/scripts/cat_error.py
    ./sql/hive/src/test/resources/data/scripts/doubleescapedtab.py
    ./sql/hive/src/test/resources/data/scripts/dumpdata_script.py
    ./sql/hive/src/test/resources/data/scripts/escapedcarriagereturn.py
    ./sql/hive/src/test/resources/data/scripts/escapednewline.py
    ./sql/hive/src/test/resources/data/scripts/escapedtab.py
    ./sql/hive/src/test/resources/data/scripts/input20_script.py
    ./sql/hive/src/test/resources/data/scripts/newline.py
    ```
    
    ## How was this patch tested?
    
    - `./python/docs/epytext.py`
    
      ```bash
      cd ./python/docs $$ make html
      ```
    
    - pep8 check (Python 2.7 / Python 3.3.6)
    
      ```
      ./dev/lint-python
      ```
    
    - `./dev/merge_spark_pr.py` (Python 2.7 only / Python 3.3.6 not working)
    
      ```bash
      python -m doctest -v ./dev/merge_spark_pr.py
      ```
    
    - `./dev/create-release/releaseutils.py` `./dev/create-release/generate-contributors.py` `./dev/create-release/translate-contributors.py` (Python 2.7 only / Python 3.3.6 not working)
    
      ```bash
      python generate-contributors.py
      python translate-contributors.py
      ```
    
    - Examples (Python 2.7 / Python 3.3.6)
    
      ```bash
      ./bin/spark-submit examples/src/main/python/mllib/decision_tree_classification_example.py
      ./bin/spark-submit examples/src/main/python/mllib/decision_tree_regression_example.py
      ./bin/spark-submit examples/src/main/python/mllib/gradient_boosting_classification_example.py
      ./bin/spark-submit examples/src/main/python/mllib/gradient_boosting_regression_example.p
      ./bin/spark-submit examples/src/main/python/mllib/random_forest_classification_example.py
      ./bin/spark-submit examples/src/main/python/mllib/random_forest_regression_example.py
      ```
    
    - Examples (Python 2.7 only / Python 3.3.6 not working)
      ```
      ./bin/spark-submit examples/src/main/python/mllib/linear_regression_with_sgd_example.py
      ./bin/spark-submit examples/src/main/python/mllib/logistic_regression_with_lbfgs_example.py
      ./bin/spark-submit examples/src/main/python/mllib/naive_bayes_example.py
      ./bin/spark-submit examples/src/main/python/mllib/svm_with_sgd_example.py
      ```
    
    - `sql/hive/src/test/resources/data/scripts/*.py` (Python 2.7 / Python 3.3.6 within suggested changes)
    
      Manually tested only changed ones.
    
    - `./dev/github_jira_sync.py` (Python 2.7 only / Python 3.3.6 not working)
    
      Manually tested this after disabling actually adding comments and links.
    
    And also via Jenkins tests.
    
    Author: hyukjinkwon <gurwls223@gmail.com>
    
    Closes #16405 from HyukjinKwon/minor-pep8.