Skip to content
Snippets Groups Projects
Commit 803e7f08 authored by Marcelo Vanzin's avatar Marcelo Vanzin Committed by Thomas Graves
Browse files

[SPARK-3979] [yarn] Use fs's default replication.

This avoids issues when HDFS is configured in a way that would not
allow the hardcoded default replication of "3".

Note: getDefaultReplication(Path) was added in 0.23.3, and the oldest
one available on Maven Central is 0.23.7, so I chose to not add code
to access that method via reflection.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #2831 from vanzin/SPARK-3979 and squashes the following commits:

b0e3a97 [Marcelo Vanzin] [SPARK-3979] [yarn] Use fs's default replication.
parent c3518620
No related branches found
No related tags found
No related merge requests found
......@@ -143,7 +143,8 @@ private[spark] trait ClientBase extends Logging {
val nns = getNameNodesToAccess(sparkConf) + dst
obtainTokensForNamenodes(nns, hadoopConf, credentials)
val replication = sparkConf.getInt("spark.yarn.submit.file.replication", 3).toShort
val replication = sparkConf.getInt("spark.yarn.submit.file.replication",
fs.getDefaultReplication(dst)).toShort
val localResources = HashMap[String, LocalResource]()
FileSystem.mkdirs(fs, dst, new FsPermission(STAGING_DIR_PERMISSION))
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment