Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    ac6fe67e
    [SPARK-5363] [PySpark] check ending mark in non-block way · ac6fe67e
    Davies Liu authored
    There is chance of dead lock that the Python process is waiting for ending mark from JVM, but which is eaten by corrupted stream.
    
    This PR checks the ending mark from Python in non-block way, so it will not blocked by Python process.
    
    There is a small chance that the ending mark is sent by Python process but not available right now, then Python process will not be used.
    
    cc JoshRosen pwendell
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #4601 from davies/freeze and squashes the following commits:
    
    e15a8c3 [Davies Liu] update logging
    890329c [Davies Liu] Merge branch 'freeze' of github.com:davies/spark into freeze
    2bd2228 [Davies Liu] add more logging
    656d544 [Davies Liu] Update PythonRDD.scala
    05e1085 [Davies Liu] check ending mark in non-block way
    ac6fe67e
    History
    [SPARK-5363] [PySpark] check ending mark in non-block way
    Davies Liu authored
    There is chance of dead lock that the Python process is waiting for ending mark from JVM, but which is eaten by corrupted stream.
    
    This PR checks the ending mark from Python in non-block way, so it will not blocked by Python process.
    
    There is a small chance that the ending mark is sent by Python process but not available right now, then Python process will not be used.
    
    cc JoshRosen pwendell
    
    Author: Davies Liu <davies@databricks.com>
    
    Closes #4601 from davies/freeze and squashes the following commits:
    
    e15a8c3 [Davies Liu] update logging
    890329c [Davies Liu] Merge branch 'freeze' of github.com:davies/spark into freeze
    2bd2228 [Davies Liu] add more logging
    656d544 [Davies Liu] Update PythonRDD.scala
    05e1085 [Davies Liu] check ending mark in non-block way