Skip to content
Snippets Groups Projects
  • David Tolpin's avatar
    437583f6
    [SPARK-11904][PYSPARK] reduceByKeyAndWindow does not require checkpointing when invFunc is None · 437583f6
    David Tolpin authored
    when invFunc is None, `reduceByKeyAndWindow(func, None, winsize, slidesize)` is equivalent to
    
         reduceByKey(func).window(winsize, slidesize).reduceByKey(winsize, slidesize)
    
    and no checkpoint is necessary. The corresponding Scala code does exactly that, but Python code always creates a windowed stream with obligatory checkpointing. The patch fixes this.
    
    I do not know how to unit-test this.
    
    Author: David Tolpin <david.tolpin@gmail.com>
    
    Closes #9888 from dtolpin/master.
    437583f6
    History
    [SPARK-11904][PYSPARK] reduceByKeyAndWindow does not require checkpointing when invFunc is None
    David Tolpin authored
    when invFunc is None, `reduceByKeyAndWindow(func, None, winsize, slidesize)` is equivalent to
    
         reduceByKey(func).window(winsize, slidesize).reduceByKey(winsize, slidesize)
    
    and no checkpoint is necessary. The corresponding Scala code does exactly that, but Python code always creates a windowed stream with obligatory checkpointing. The patch fixes this.
    
    I do not know how to unit-test this.
    
    Author: David Tolpin <david.tolpin@gmail.com>
    
    Closes #9888 from dtolpin/master.