Skip to content
Snippets Groups Projects
  • Bryan Cutler's avatar
    77cc0d67
    [SPARK-12717][PYTHON] Adding thread-safe broadcast pickle registry · 77cc0d67
    Bryan Cutler authored
    ## What changes were proposed in this pull request?
    
    When using PySpark broadcast variables in a multi-threaded environment,  `SparkContext._pickled_broadcast_vars` becomes a shared resource.  A race condition can occur when broadcast variables that are pickled from one thread get added to the shared ` _pickled_broadcast_vars` and become part of the python command from another thread.  This PR introduces a thread-safe pickled registry using thread local storage so that when python command is pickled (causing the broadcast variable to be pickled and added to the registry) each thread will have their own view of the pickle registry to retrieve and clear the broadcast variables used.
    
    ## How was this patch tested?
    
    Added a unit test that causes this race condition using another thread.
    
    Author: Bryan Cutler <cutlerb@gmail.com>
    
    Closes #18695 from BryanCutler/pyspark-bcast-threadsafe-SPARK-12717.
    77cc0d67
    History
    [SPARK-12717][PYTHON] Adding thread-safe broadcast pickle registry
    Bryan Cutler authored
    ## What changes were proposed in this pull request?
    
    When using PySpark broadcast variables in a multi-threaded environment,  `SparkContext._pickled_broadcast_vars` becomes a shared resource.  A race condition can occur when broadcast variables that are pickled from one thread get added to the shared ` _pickled_broadcast_vars` and become part of the python command from another thread.  This PR introduces a thread-safe pickled registry using thread local storage so that when python command is pickled (causing the broadcast variable to be pickled and added to the registry) each thread will have their own view of the pickle registry to retrieve and clear the broadcast variables used.
    
    ## How was this patch tested?
    
    Added a unit test that causes this race condition using another thread.
    
    Author: Bryan Cutler <cutlerb@gmail.com>
    
    Closes #18695 from BryanCutler/pyspark-bcast-threadsafe-SPARK-12717.
broadcast.py 5.02 KiB