Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    e595c8d0
    [SPARK-3993] [PySpark] fix bug while reuse worker after take() · e595c8d0
    Davies Liu authored
    After take(), maybe there are some garbage left in the socket, then next task assigned to this worker will hang because of corrupted data.
    
    We should make sure the socket is clean before reuse it, write END_OF_STREAM at the end, and check it after read out all result from python.
    
    Author: Davies Liu <davies.liu@gmail.com>
    Author: Davies Liu <davies@databricks.com>
    
    Closes #2838 from davies/fix_reuse and squashes the following commits:
    
    8872914 [Davies Liu] fix tests
    660875b [Davies Liu] fix bug while reuse worker after take()
    e595c8d0
    History
    [SPARK-3993] [PySpark] fix bug while reuse worker after take()
    Davies Liu authored
    After take(), maybe there are some garbage left in the socket, then next task assigned to this worker will hang because of corrupted data.
    
    We should make sure the socket is clean before reuse it, write END_OF_STREAM at the end, and check it after read out all result from python.
    
    Author: Davies Liu <davies.liu@gmail.com>
    Author: Davies Liu <davies@databricks.com>
    
    Closes #2838 from davies/fix_reuse and squashes the following commits:
    
    8872914 [Davies Liu] fix tests
    660875b [Davies Liu] fix bug while reuse worker after take()
worker.py 5.25 KiB