-
- Downloads
[SPARK-16736][CORE][SQL] purge superfluous fs calls
A review of the code, working back from Hadoop's `FileSystem.exists()` and `FileSystem.isDirectory()` code, then removing uses of the calls when superfluous. 1. delete is harmless if called on a nonexistent path, so don't do any checks before deletes 1. any `FileSystem.exists()` check before `getFileStatus()` or `open()` is superfluous as the operation itself does the check. Instead the `FileNotFoundException` is caught and triggers the downgraded path. When a `FileNotFoundException` was thrown before, the code still creates a new FNFE with the error messages. Though now the inner exceptions are nested, for easier diagnostics. Initially, relying on Jenkins test runs. One troublespot here is that some of the codepaths are clearly error situations; it's not clear that they have coverage anyway. Trying to create the failure conditions in tests would be ideal, but it will also be hard. Author: Steve Loughran <stevel@apache.org> Closes #14371 from steveloughran/cloud/SPARK-16736-superfluous-fs-calls.
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 0 additions, 3 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala 12 additions, 15 deletions...a/org/apache/spark/deploy/history/FsHistoryProvider.scala
- core/src/main/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala 13 additions, 18 deletions...in/scala/org/apache/spark/rdd/ReliableCheckpointRDD.scala
- core/src/main/scala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala 1 addition, 6 deletions...cala/org/apache/spark/rdd/ReliableRDDCheckpointData.scala
- core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala 2 additions, 11 deletions...ala/org/apache/spark/scheduler/EventLoggingListener.scala
- repl/src/main/scala/org/apache/spark/repl/ExecutorClassLoader.scala 5 additions, 4 deletions...ain/scala/org/apache/spark/repl/ExecutorClassLoader.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala 17 additions, 15 deletions...cution/streaming/state/HDFSBackedStateStoreProvider.scala
- sql/hive/src/test/java/org/apache/spark/sql/hive/JavaMetastoreDataSourcesSuite.java 1 addition, 3 deletions.../apache/spark/sql/hive/JavaMetastoreDataSourcesSuite.java
- sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala 1 addition, 1 deletion...org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
- streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala 7 additions, 10 deletions...rc/main/scala/org/apache/spark/streaming/Checkpoint.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/FileBasedWriteAheadLog.scala 20 additions, 7 deletions.../apache/spark/streaming/util/FileBasedWriteAheadLog.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/HdfsUtils.scala 11 additions, 13 deletions...ain/scala/org/apache/spark/streaming/util/HdfsUtils.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala 2 additions, 3 deletions.../src/main/scala/org/apache/spark/deploy/yarn/Client.scala
Loading
Please register or sign in to comment