Skip to content
Snippets Groups Projects
  • Ahir Reddy's avatar
    7b6d7748
    [SPARK-986]: Job cancelation for PySpark · 7b6d7748
    Ahir Reddy authored
    
    * Additions to the PySpark API to cancel jobs
    * Monitor Thread in PythonRDD to kill Python workers if a task is interrupted
    
    Author: Ahir Reddy <ahirreddy@gmail.com>
    
    Closes #541 from ahirreddy/python-cancel and squashes the following commits:
    
    dfdf447 [Ahir Reddy] Changed success -> completed and made logging message clearer
    6c860ab [Ahir Reddy] PR Comments
    4b4100a [Ahir Reddy] Success flag
    adba6ed [Ahir Reddy] Destroy python workers
    27a2f8f [Ahir Reddy] Start the writer thread...
    d422f7b [Ahir Reddy] Remove unnecesssary vals
    adda337 [Ahir Reddy] Busy wait on the ocntext.interrupted flag, and then kill the python worker
    d9e472f [Ahir Reddy] Revert "removed unnecessary vals"
    5b9cae5 [Ahir Reddy] removed unnecessary vals
    07b54d9 [Ahir Reddy] Fix canceling unit test
    8ae9681 [Ahir Reddy] Don't interrupt worker
    7722342 [Ahir Reddy] Monitor Thread for python workers
    db04e16 [Ahir Reddy] Added canceling api to PySpark
    
    (cherry picked from commit e53eb4f0)
    Signed-off-by: default avatarMatei Zaharia <matei@databricks.com>
    7b6d7748
    History
    [SPARK-986]: Job cancelation for PySpark
    Ahir Reddy authored
    
    * Additions to the PySpark API to cancel jobs
    * Monitor Thread in PythonRDD to kill Python workers if a task is interrupted
    
    Author: Ahir Reddy <ahirreddy@gmail.com>
    
    Closes #541 from ahirreddy/python-cancel and squashes the following commits:
    
    dfdf447 [Ahir Reddy] Changed success -> completed and made logging message clearer
    6c860ab [Ahir Reddy] PR Comments
    4b4100a [Ahir Reddy] Success flag
    adba6ed [Ahir Reddy] Destroy python workers
    27a2f8f [Ahir Reddy] Start the writer thread...
    d422f7b [Ahir Reddy] Remove unnecesssary vals
    adda337 [Ahir Reddy] Busy wait on the ocntext.interrupted flag, and then kill the python worker
    d9e472f [Ahir Reddy] Revert "removed unnecessary vals"
    5b9cae5 [Ahir Reddy] removed unnecessary vals
    07b54d9 [Ahir Reddy] Fix canceling unit test
    8ae9681 [Ahir Reddy] Don't interrupt worker
    7722342 [Ahir Reddy] Monitor Thread for python workers
    db04e16 [Ahir Reddy] Added canceling api to PySpark
    
    (cherry picked from commit e53eb4f0)
    Signed-off-by: default avatarMatei Zaharia <matei@databricks.com>