Skip to content
Snippets Groups Projects
  • cody koeninger's avatar
    10423258
    [SPARK-17813][SQL][KAFKA] Maximum data per trigger · 10423258
    cody koeninger authored
    ## What changes were proposed in this pull request?
    
    maxOffsetsPerTrigger option for rate limiting, proportionally based on volume of different topicpartitions.
    
    ## How was this patch tested?
    
    Added unit test
    
    Author: cody koeninger <cody@koeninger.org>
    
    Closes #15527 from koeninger/SPARK-17813.
    10423258
    History
    [SPARK-17813][SQL][KAFKA] Maximum data per trigger
    cody koeninger authored
    ## What changes were proposed in this pull request?
    
    maxOffsetsPerTrigger option for rate limiting, proportionally based on volume of different topicpartitions.
    
    ## How was this patch tested?
    
    Added unit test
    
    Author: cody koeninger <cody@koeninger.org>
    
    Closes #15527 from koeninger/SPARK-17813.