Skip to content
  • Joseph K. Bradley's avatar
    1db7179f
    [SPARK-9805] [MLLIB] [PYTHON] [STREAMING] Added _eventually for ml streaming pyspark tests · 1db7179f
    Joseph K. Bradley authored
    Recently, PySpark ML streaming tests have been flaky, most likely because of the batches not being processed in time.  Proposal: Replace the use of _ssc_wait (which waits for a fixed amount of time) with a method which waits for a fixed amount of time but can terminate early based on a termination condition method.  With this, we can extend the waiting period (to make tests less flaky) but also stop early when possible (making tests faster on average, which I verified locally).
    
    CC: mengxr tdas freeman-lab
    
    Author: Joseph K. Bradley <joseph@databricks.com>
    
    Closes #8087 from jkbradley/streaming-ml-tests.
    1db7179f
    [SPARK-9805] [MLLIB] [PYTHON] [STREAMING] Added _eventually for ml streaming pyspark tests
    Joseph K. Bradley authored
    Recently, PySpark ML streaming tests have been flaky, most likely because of the batches not being processed in time.  Proposal: Replace the use of _ssc_wait (which waits for a fixed amount of time) with a method which waits for a fixed amount of time but can terminate early based on a termination condition method.  With this, we can extend the waiting period (to make tests less flaky) but also stop early when possible (making tests faster on average, which I verified locally).
    
    CC: mengxr tdas freeman-lab
    
    Author: Joseph K. Bradley <joseph@databricks.com>
    
    Closes #8087 from jkbradley/streaming-ml-tests.
Loading