python/pyspark/sql/streaming.py · 3d75a5b2a76eba0855d73476dc2fd579c612d521 · cs525-sp18-g07 / spark

An error occurred while loading the file. Please try again.

9 years ago

[SPARK-16313][SQL] Spark should not silently drop exceptions in file listing · 3d75a5b2

Reynold Xin authored 9 years ago

## What changes were proposed in this pull request?
Spark silently drops exceptions during file listing. This is a very bad behavior because it can mask legitimate errors and the resulting plan will silently have 0 rows. This patch changes it to not silently drop the errors.

## How was this patch tested?
Manually verified.

Author: Reynold Xin <rxin@databricks.com>

Closes #13987 from rxin/SPARK-16313.

3d75a5b2

History

[SPARK-16313][SQL] Spark should not silently drop exceptions in file listing

Reynold Xin authored 9 years ago

## What changes were proposed in this pull request?
Spark silently drops exceptions during file listing. This is a very bad behavior because it can mask legitimate errors and the resulting plan will silently have 0 rows. This patch changes it to not silently drop the errors.

## How was this patch tested?
Manually verified.

Author: Reynold Xin <rxin@databricks.com>

Closes #13987 from rxin/SPARK-16313.