-
- Downloads
[SPARK-8652] [PYSPARK] Check return value for all uses of doctest.testmod()
This patch addresses a critical issue in the PySpark tests: Several of our Python modules' `__main__` methods call `doctest.testmod()` in order to run doctests but forget to check and handle its return value. As a result, some PySpark test failures can go unnoticed because they will not fail the build. Fortunately, there was only one test failure which was masked by this bug: a `pyspark.profiler` doctest was failing due to changes in RDD pipelining. Author: Josh Rosen <joshrosen@databricks.com> Closes #7032 from JoshRosen/testmod-fix and squashes the following commits: 60dbdc0 [Josh Rosen] Account for int vs. long formatting change in Python 3 8b8d80a [Josh Rosen] Fix failing test. e6423f9 [Josh Rosen] Check return code for all uses of doctest.testmod().
Showing
- dev/merge_spark_pr.py 3 additions, 1 deletiondev/merge_spark_pr.py
- python/pyspark/accumulators.py 3 additions, 1 deletionpython/pyspark/accumulators.py
- python/pyspark/broadcast.py 3 additions, 1 deletionpython/pyspark/broadcast.py
- python/pyspark/heapq3.py 3 additions, 2 deletionspython/pyspark/heapq3.py
- python/pyspark/profiler.py 6 additions, 2 deletionspython/pyspark/profiler.py
- python/pyspark/serializers.py 5 additions, 3 deletionspython/pyspark/serializers.py
- python/pyspark/shuffle.py 3 additions, 1 deletionpython/pyspark/shuffle.py
- python/pyspark/streaming/util.py 3 additions, 1 deletionpython/pyspark/streaming/util.py
Loading
Please register or sign in to comment