Skip to content
Snippets Groups Projects
Commit 5ad4e32d authored by Takeshi YAMAMURO's avatar Takeshi YAMAMURO Committed by Yin Huai
Browse files

[SPARK-15530][SQL] Set #parallelism for file listing in listLeafFilesInParallel 

## What changes were proposed in this pull request?
This pr is to set the number of parallelism to prevent file listing in `listLeafFilesInParallel` from generating many tasks in case of large #defaultParallelism.

## How was this patch tested?
Manually checked

Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>

Closes #13444 from maropu/SPARK-15530.
parent 3b7fb84c
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment