Skip to content
Snippets Groups Projects
Commit 6b2baec0 authored by Kay Ousterhout's avatar Kay Ousterhout Committed by Josh Rosen
Browse files

[SPARK-9326] Close lock file used for file downloads.

A lock file is used to ensure multiple executors running on the
same machine don't download the same file concurrently. Spark never
closes these lock files (releasing the lock does not close the
underlying file); this commit fixes that.

cc vanzin (looks like you've been involved in various other fixes surrounding these lock files)

Author: Kay Ousterhout <kayousterhout@gmail.com>

Closes #7650 from kayousterhout/SPARK-9326 and squashes the following commits:

0401bd1 [Kay Ousterhout] Close lock file used for file downloads.
parent 1cf19760
No related branches found
No related tags found
No related merge requests found
......@@ -443,11 +443,11 @@ private[spark] object Utils extends Logging {
val lockFileName = s"${url.hashCode}${timestamp}_lock"
val localDir = new File(getLocalDir(conf))
val lockFile = new File(localDir, lockFileName)
val raf = new RandomAccessFile(lockFile, "rw")
val lockFileChannel = new RandomAccessFile(lockFile, "rw").getChannel()
// Only one executor entry.
// The FileLock is only used to control synchronization for executors download file,
// it's always safe regardless of lock type (mandatory or advisory).
val lock = raf.getChannel().lock()
val lock = lockFileChannel.lock()
val cachedFile = new File(localDir, cachedFileName)
try {
if (!cachedFile.exists()) {
......@@ -455,6 +455,7 @@ private[spark] object Utils extends Logging {
}
} finally {
lock.release()
lockFileChannel.close()
}
copyFile(
url,
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment