Skip to content
Snippets Groups Projects
  • Reynold Xin's avatar
    d601894c
    [SPARK-16335][SQL] Structured streaming should fail if source directory does not exist · d601894c
    Reynold Xin authored
    ## What changes were proposed in this pull request?
    In structured streaming, Spark does not report errors when the specified directory does not exist. This is a behavior different from the batch mode. This patch changes the behavior to fail if the directory does not exist (when the path is not a glob pattern).
    
    ## How was this patch tested?
    Updated unit tests to reflect the new behavior.
    
    Author: Reynold Xin <rxin@databricks.com>
    
    Closes #14002 from rxin/SPARK-16335.
    d601894c
    History
    [SPARK-16335][SQL] Structured streaming should fail if source directory does not exist
    Reynold Xin authored
    ## What changes were proposed in this pull request?
    In structured streaming, Spark does not report errors when the specified directory does not exist. This is a behavior different from the batch mode. This patch changes the behavior to fail if the directory does not exist (when the path is not a glob pattern).
    
    ## How was this patch tested?
    Updated unit tests to reflect the new behavior.
    
    Author: Reynold Xin <rxin@databricks.com>
    
    Closes #14002 from rxin/SPARK-16335.