Skip to content
Snippets Groups Projects
  • Mark Grover's avatar
    55b8cfe6
    [SPARK-19185][DSTREAM] Make Kafka consumer cache configurable · 55b8cfe6
    Mark Grover authored
    ## What changes were proposed in this pull request?
    
    Add a new property `spark.streaming.kafka.consumer.cache.enabled` that allows users to enable or disable the cache for Kafka consumers. This property can be especially handy in cases where issues like SPARK-19185 get hit, for which there isn't a solution committed yet. By default, the cache is still on, so this change doesn't change any out-of-box behavior.
    
    ## How was this patch tested?
    Running unit tests
    
    Author: Mark Grover <mark@apache.org>
    Author: Mark Grover <grover.markgrover@gmail.com>
    
    Closes #18234 from markgrover/spark-19185.
    55b8cfe6
    History
    [SPARK-19185][DSTREAM] Make Kafka consumer cache configurable
    Mark Grover authored
    ## What changes were proposed in this pull request?
    
    Add a new property `spark.streaming.kafka.consumer.cache.enabled` that allows users to enable or disable the cache for Kafka consumers. This property can be especially handy in cases where issues like SPARK-19185 get hit, for which there isn't a solution committed yet. By default, the cache is still on, so this change doesn't change any out-of-box behavior.
    
    ## How was this patch tested?
    Running unit tests
    
    Author: Mark Grover <mark@apache.org>
    Author: Mark Grover <grover.markgrover@gmail.com>
    
    Closes #18234 from markgrover/spark-19185.