-
- Downloads
[SPARK-17676][CORE] FsHistoryProvider should ignore hidden files
## What changes were proposed in this pull request? FsHistoryProvider was writing a hidden file (to check the fs's clock). Even though it deleted the file immediately, sometimes another thread would try to scan the files on the fs in-between, and then there would be an error msg logged which was very misleading for the end-user. (The logged error was harmless, though.) ## How was this patch tested? I added one unit test, but to be clear, that test was passing before. The actual change in behavior in that test is just logging (after the change, there is no more logged error), which I just manually verified. Author: Imran Rashid <irashid@cloudera.com> Closes #15250 from squito/SPARK-17676.
Showing
- core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala 6 additions, 1 deletion...a/org/apache/spark/deploy/history/FsHistoryProvider.scala
- core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala 34 additions, 2 deletions.../apache/spark/deploy/history/FsHistoryProviderSuite.scala
Loading
Please register or sign in to comment