Skip to content
Snippets Groups Projects
Commit 078f3fbd authored by Chia-Yung Su's avatar Chia-Yung Su Committed by Michael Armbrust
Browse files

[SPARK-3011][SQL] _temporary directory should be filtered out by sqlContext.parquetFile

Author: Chia-Yung Su <chiayung@appier.com>

Closes #1924 from joesu/bugfix-spark3011 and squashes the following commits:

c7e44f2 [Chia-Yung Su] match syntax
f8fc32a [Chia-Yung Su] filter out tmp dir
parent 6b8de0e3
No related branches found
No related tags found
No related merge requests found
......@@ -375,7 +375,8 @@ private[parquet] object ParquetTypesConverter extends Logging {
val children = fs.listStatus(path).filterNot { status =>
val name = status.getPath.getName
name(0) == '.' || name == FileOutputCommitter.SUCCEEDED_FILE_NAME
name(0) == '.' || name == FileOutputCommitter.SUCCEEDED_FILE_NAME ||
name == FileOutputCommitter.TEMP_DIR_NAME
}
// NOTE (lian): Parquet "_metadata" file can be very slow if the file consists of lots of row
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment