Skip to content
Snippets Groups Projects
Commit ab2abcb5 authored by Christophe Préaud's avatar Christophe Préaud Committed by Josh Rosen
Browse files

[SPARK-4764] Ensure that files are fetched atomically

tempFile is created in the same directory than targetFile, so that the
move from tempFile to targetFile is always atomic

Author: Christophe Préaud <christophe.preaud@kelkoo.com>

Closes #2855 from preaudc/master and squashes the following commits:

9ba89ca [Christophe Préaud] Ensure that files are fetched atomically
54419ae [Christophe Préaud] Merge remote-tracking branch 'upstream/master'
c6a5590 [Christophe Préaud] Revert commit 8ea871f8130b2490f1bad7374a819bf56f0ccbbd
7456a33 [Christophe Préaud] Merge remote-tracking branch 'upstream/master'
8ea871f [Christophe Préaud] Ensure that files are fetched atomically
parent 8817fc7f
No related branches found
No related tags found
No related merge requests found
...@@ -425,8 +425,7 @@ private[spark] object Utils extends Logging { ...@@ -425,8 +425,7 @@ private[spark] object Utils extends Logging {
conf: SparkConf, conf: SparkConf,
securityMgr: SecurityManager, securityMgr: SecurityManager,
hadoopConf: Configuration) { hadoopConf: Configuration) {
val tempDir = getLocalDir(conf) val tempFile = File.createTempFile("fetchFileTemp", null, new File(targetDir.getAbsolutePath))
val tempFile = File.createTempFile("fetchFileTemp", null, new File(tempDir))
val targetFile = new File(targetDir, filename) val targetFile = new File(targetDir, filename)
val uri = new URI(url) val uri = new URI(url)
val fileOverwrite = conf.getBoolean("spark.files.overwrite", defaultValue = false) val fileOverwrite = conf.getBoolean("spark.files.overwrite", defaultValue = false)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment