-
- Downloads
[SPARK-16414][YARN] Fix bugs for "Can not get user config when calling...
[SPARK-16414][YARN] Fix bugs for "Can not get user config when calling SparkHadoopUtil.get.conf on yarn cluser mode" ## What changes were proposed in this pull request? The `SparkHadoopUtil` singleton was instantiated before `ApplicationMaster` in `ApplicationMaster.main` when deploying spark on yarn cluster mode, the `conf` in the `SparkHadoopUtil` singleton didn't include user's configuration. So, we should load the properties file with the Spark configuration and set entries as system properties before `SparkHadoopUtil` first instantiate. ## How was this patch tested? Add a test case Author: sharkd <sharkd.tu@gmail.com> Author: sharkdtu <sharkdtu@tencent.com> Closes #14088 from sharkdtu/master.
Showing
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 9 additions, 8 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
- yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala 45 additions, 0 deletions...scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala
Loading
Please register or sign in to comment