Skip to content
Snippets Groups Projects
Commit 02e45729 authored by Gera Shegalov's avatar Gera Shegalov Committed by Thomas Graves
Browse files

[YARN] SPARK-2577: File upload to viewfs is broken due to mount point re...

Opting to the option 2 defined in SPARK-2577, i.e., retrieve and pass the correct file system object to addResource.

Author: Gera Shegalov <gera@twitter.com>

Closes #1483 from gerashegalov/master and squashes the following commits:

90c9087 [Gera Shegalov] [YARN] SPARK-2577: File upload to viewfs is broken due to mount point resolution
parent ddadf1b0
No related branches found
No related tags found
No related merge requests found
......@@ -232,7 +232,8 @@ trait ClientBase extends Logging {
if (!ClientBase.LOCAL_SCHEME.equals(localURI.getScheme())) {
val setPermissions = if (destName.equals(ClientBase.APP_JAR)) true else false
val destPath = copyRemoteFile(dst, qualifyForLocal(localURI), replication, setPermissions)
distCacheMgr.addResource(fs, conf, destPath, localResources, LocalResourceType.FILE,
val destFs = FileSystem.get(destPath.toUri(), conf)
distCacheMgr.addResource(destFs, conf, destPath, localResources, LocalResourceType.FILE,
destName, statCache)
} else if (confKey != null) {
sparkConf.set(confKey, localPath)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment