-
- Downloads
[SPARK-14109][SQL] Fix HDFSMetadataLog to fallback from FileContext to FileSystem API
## What changes were proposed in this pull request? HDFSMetadataLog uses newer FileContext API to achieve atomic renaming. However, FileContext implementations may not exist for many scheme for which there may be FileSystem implementations. In those cases, rather than failing completely, we should fallback to the FileSystem based implementation, and log warning that there may be file consistency issues in case the log directory is concurrently modified. In addition I have also added more tests to increase the code coverage. ## How was this patch tested? Unit test. Tested on cluster with custom file system. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #11925 from tdas/SPARK-14109.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala 161 additions, 18 deletions...pache/spark/sql/execution/streaming/HDFSMetadataLog.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLogSuite.scala 117 additions, 5 deletions.../spark/sql/execution/streaming/HDFSMetadataLogSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/SharedSQLContext.scala 4 additions, 1 deletion...st/scala/org/apache/spark/sql/test/SharedSQLContext.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala 6 additions, 2 deletions...test/scala/org/apache/spark/sql/test/TestSQLContext.scala
Loading
Please register or sign in to comment