Skip to content
Snippets Groups Projects
Commit 41d698ec authored by Eric Liang's avatar Eric Liang Committed by Wenchen Fan
Browse files

[SPARK-18661][SQL] Creating a partitioned datasource table should not scan all files for table


## What changes were proposed in this pull request?

Even though in 2.1 creating a partitioned datasource table will not populate the partition data by default (until the user issues MSCK REPAIR TABLE), it seems we still scan the filesystem for no good reason.

We should avoid doing this when the user specifies a schema.

## How was this patch tested?

Perf stat tests.

Author: Eric Liang <ekl@databricks.com>

Closes #16090 from ericl/spark-18661.

(cherry picked from commit d9eb4c72)
Signed-off-by: default avatarWenchen Fan <wenchen@databricks.com>
parent 8145c82b
No related branches found
No related tags found
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment