Skip to content
Snippets Groups Projects
Commit 3c9802d9 authored by Hao Zhu's avatar Hao Zhu Committed by Tathagata Das
Browse files

[SPARK-9801] [STREAMING] Check if file exists before deleting temporary files.

Spark streaming deletes the temp file and backup files without checking if they exist or not

Author: Hao Zhu <viadeazhu@gmail.com>

Closes #8082 from viadea/master and squashes the following commits:

242d05f [Hao Zhu] [SPARK-9801][Streaming]No need to check the existence of those files
fd143f2 [Hao Zhu] [SPARK-9801][Streaming]Check if backupFile exists before deleting backupFile files.
087daf0 [Hao Zhu] SPARK-9801
parent 853809e9
No related branches found
No related tags found
No related merge requests found
......@@ -192,7 +192,9 @@ class CheckpointWriter(
+ "'")
// Write checkpoint to temp file
fs.delete(tempFile, true) // just in case it exists
if (fs.exists(tempFile)) {
fs.delete(tempFile, true) // just in case it exists
}
val fos = fs.create(tempFile)
Utils.tryWithSafeFinally {
fos.write(bytes)
......@@ -203,7 +205,9 @@ class CheckpointWriter(
// If the checkpoint file exists, back it up
// If the backup exists as well, just delete it, otherwise rename will fail
if (fs.exists(checkpointFile)) {
fs.delete(backupFile, true) // just in case it exists
if (fs.exists(backupFile)){
fs.delete(backupFile, true) // just in case it exists
}
if (!fs.rename(checkpointFile, backupFile)) {
logWarning("Could not rename " + checkpointFile + " to " + backupFile)
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment