Skip to content
Snippets Groups Projects
  • David Tolpin's avatar
    599a8c6e
    [SPARK-11812][PYSPARK] invFunc=None works properly with python's reduceByKeyAndWindow · 599a8c6e
    David Tolpin authored
    invFunc is optional and can be None. Instead of invFunc (the parameter) invReduceFunc (a local function) was checked for trueness (that is, not None, in this context). A local function is never None,
    thus the case of invFunc=None (a common one when inverse reduction is not defined) was treated incorrectly, resulting in loss of data.
    
    In addition, the docstring used wrong parameter names, also fixed.
    
    Author: David Tolpin <david.tolpin@gmail.com>
    
    Closes #9775 from dtolpin/master.
    599a8c6e
    History
    [SPARK-11812][PYSPARK] invFunc=None works properly with python's reduceByKeyAndWindow
    David Tolpin authored
    invFunc is optional and can be None. Instead of invFunc (the parameter) invReduceFunc (a local function) was checked for trueness (that is, not None, in this context). A local function is never None,
    thus the case of invFunc=None (a common one when inverse reduction is not defined) was treated incorrectly, resulting in loss of data.
    
    In addition, the docstring used wrong parameter names, also fixed.
    
    Author: David Tolpin <david.tolpin@gmail.com>
    
    Closes #9775 from dtolpin/master.
tests.py 57.86 KiB