-
- Downloads
[SPARK-529][CORE][YARN] Add type-safe config keys to SparkConf.
This is, in a way, the basics to enable SPARK-529 (which was closed as won't fix but I think is still valuable). In fact, Spark SQL created something for that, and this change basically factors out that code and inserts it into SparkConf, with some extra bells and whistles. To showcase the usage of this pattern, I modified the YARN backend to use the new config keys (defined in the new `config` package object under `o.a.s.deploy.yarn`). Most of the changes are mechanic, although logic had to be slightly modified in a handful of places. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #10205 from vanzin/conf-opts.
Showing
- common/network-common/src/main/java/org/apache/spark/network/util/JavaUtils.java 12 additions, 13 deletions...rc/main/java/org/apache/spark/network/util/JavaUtils.java
- core/src/main/scala/org/apache/spark/SparkConf.scala 38 additions, 1 deletioncore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala 184 additions, 0 deletions...cala/org/apache/spark/internal/config/ConfigBuilder.scala
- core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala 111 additions, 0 deletions.../scala/org/apache/spark/internal/config/ConfigEntry.scala
- core/src/main/scala/org/apache/spark/internal/config/package.scala 76 additions, 0 deletions...main/scala/org/apache/spark/internal/config/package.scala
- core/src/test/scala/org/apache/spark/internal/config/ConfigEntrySuite.scala 155 additions, 0 deletions...a/org/apache/spark/internal/config/ConfigEntrySuite.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala 7 additions, 7 deletions...g/apache/spark/deploy/yarn/AMDelegationTokenRenewer.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 14 additions, 14 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala 102 additions, 128 deletions.../src/main/scala/org/apache/spark/deploy/yarn/Client.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala 24 additions, 29 deletions.../scala/org/apache/spark/deploy/yarn/ClientArguments.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala 2 additions, 1 deletion...he/spark/deploy/yarn/ExecutorDelegationTokenUpdater.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala 8 additions, 6 deletions...scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/LocalityPreferredContainerPlacementStrategy.scala 2 additions, 4 deletions...oy/yarn/LocalityPreferredContainerPlacementStrategy.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala 5 additions, 5 deletions...in/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala 2 additions, 1 deletion...ain/scala/org/apache/spark/deploy/yarn/YarnRMClient.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala 8 additions, 10 deletions...la/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala 243 additions, 0 deletions.../src/main/scala/org/apache/spark/deploy/yarn/config.scala
- yarn/src/main/scala/org/apache/spark/scheduler/cluster/SchedulerExtensionService.scala 10 additions, 22 deletions...e/spark/scheduler/cluster/SchedulerExtensionService.scala
- yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala 14 additions, 12 deletions...test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
- yarn/src/test/scala/org/apache/spark/scheduler/cluster/ExtensionServiceIntegrationSuite.scala 2 additions, 2 deletions.../scheduler/cluster/ExtensionServiceIntegrationSuite.scala
Loading
Please register or sign in to comment