-
- Downloads
[SPARK-16335][SQL] Structured streaming should fail if source directory does not exist
## What changes were proposed in this pull request? In structured streaming, Spark does not report errors when the specified directory does not exist. This is a behavior different from the batch mode. This patch changes the behavior to fail if the directory does not exist (when the path is not a glob pattern). ## How was this patch tested? Updated unit tests to reflect the new behavior. Author: Reynold Xin <rxin@databricks.com> Closes #14002 from rxin/SPARK-16335.
Showing
- core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala 5 additions, 5 deletions.../main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala
- python/pyspark/sql/streaming.py 4 additions, 7 deletionspython/pyspark/sql/streaming.py
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 12 additions, 0 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala 15 additions, 23 deletions...rg/apache/spark/sql/streaming/FileStreamSourceSuite.scala
Loading
Please register or sign in to comment