Skip to content
  • Burak Yavuz's avatar
    9ee5c257
    [SPARK-14353] Dataset Time Window `window` API for Python, and SQL · 9ee5c257
    Burak Yavuz authored
    ## What changes were proposed in this pull request?
    
    The `window` function was added to Dataset with [this PR](https://github.com/apache/spark/pull/12008).
    This PR adds the Python, and SQL, API for this function.
    
    With this PR, SQL, Java, and Scala will share the same APIs as in users can use:
     - `window(timeColumn, windowDuration)`
     - `window(timeColumn, windowDuration, slideDuration)`
     - `window(timeColumn, windowDuration, slideDuration, startTime)`
    
    In Python, users can access all APIs above, but in addition they can do
     - In Python:
       `window(timeColumn, windowDuration, startTime=...)`
    
    that is, they can provide the startTime without providing the `slideDuration`. In this case, we will generate tumbling windows.
    
    ## How was this patch tested?
    
    Unit tests + manual tests
    
    Author: Burak Yavuz <brkyvz@gmail.com>
    
    Closes #12136 from brkyvz/python-windows.
    9ee5c257
    [SPARK-14353] Dataset Time Window `window` API for Python, and SQL
    Burak Yavuz authored
    ## What changes were proposed in this pull request?
    
    The `window` function was added to Dataset with [this PR](https://github.com/apache/spark/pull/12008).
    This PR adds the Python, and SQL, API for this function.
    
    With this PR, SQL, Java, and Scala will share the same APIs as in users can use:
     - `window(timeColumn, windowDuration)`
     - `window(timeColumn, windowDuration, slideDuration)`
     - `window(timeColumn, windowDuration, slideDuration, startTime)`
    
    In Python, users can access all APIs above, but in addition they can do
     - In Python:
       `window(timeColumn, windowDuration, startTime=...)`
    
    that is, they can provide the startTime without providing the `slideDuration`. In this case, we will generate tumbling windows.
    
    ## How was this patch tested?
    
    Unit tests + manual tests
    
    Author: Burak Yavuz <brkyvz@gmail.com>
    
    Closes #12136 from brkyvz/python-windows.
Loading