-
- Downloads
[SPARK-19348][PYTHON] PySpark keyword_only decorator is not thread-safe
## What changes were proposed in this pull request? The `keyword_only` decorator in PySpark is not thread-safe. It writes kwargs to a static class variable in the decorator, which is then retrieved later in the class method as `_input_kwargs`. If multiple threads are constructing the same class with different kwargs, it becomes a race condition to read from the static class variable before it's overwritten. See [SPARK-19348](https://issues.apache.org/jira/browse/SPARK-19348) for reproduction code. This change will write the kwargs to a member variable so that multiple threads can operate on separate instances without the race condition. It does not protect against multiple threads operating on a single instance, but that is better left to the user to synchronize. ## How was this patch tested? Added new unit tests for using the keyword_only decorator and a regression test that verifies `_input_kwargs` can be overwritten from different class instances. Author: Bryan Cutler <cutlerb@gmail.com> Closes #16782 from BryanCutler/pyspark-keyword_only-threadsafe-SPARK-19348.
Showing
- python/pyspark/__init__.py 6 additions, 4 deletionspython/pyspark/__init__.py
- python/pyspark/ml/classification.py 16 additions, 16 deletionspython/pyspark/ml/classification.py
- python/pyspark/ml/clustering.py 8 additions, 8 deletionspython/pyspark/ml/clustering.py
- python/pyspark/ml/evaluation.py 6 additions, 6 deletionspython/pyspark/ml/evaluation.py
- python/pyspark/ml/feature.py 60 additions, 60 deletionspython/pyspark/ml/feature.py
- python/pyspark/ml/pipeline.py 2 additions, 2 deletionspython/pyspark/ml/pipeline.py
- python/pyspark/ml/recommendation.py 2 additions, 2 deletionspython/pyspark/ml/recommendation.py
- python/pyspark/ml/regression.py 14 additions, 14 deletionspython/pyspark/ml/regression.py
- python/pyspark/ml/tests.py 4 additions, 4 deletionspython/pyspark/ml/tests.py
- python/pyspark/ml/tuning.py 4 additions, 4 deletionspython/pyspark/ml/tuning.py
- python/pyspark/tests.py 39 additions, 0 deletionspython/pyspark/tests.py
Loading
Please register or sign in to comment