-
- Downloads
Merge pull request #199 from harveyfeng/yarn-2.2
Hadoop 2.2 migration Includes support for the YARN API stabilized in the Hadoop 2.2 release, and a few style patches. Short description for each set of commits: a98f5a0e - "Misc style changes in the 'yarn' package" a67ebf43 - "A few more style fixes in the 'yarn' package" Both of these are some minor style changes, such as fixing lines over 100 chars, to the existing YARN code. ab8652f2 - "Add a 'new-yarn' directory ... " Copies everything from `SPARK_HOME/yarn` to `SPARK_HOME/new-yarn`. No actual code changes here. 4f1c3fa5 - "Hadoop 2.2 YARN API migration ..." API patches to code in the `SPARK_HOME/new-yarn` directory. There are a few more small style changes mixed in, too. Based on @colorant's Hadoop 2.2 support for the scala-2.10 branch in #141. a1a1c62a - "Add optional Hadoop 2.2 settings in sbt build ... " If Spark should be built against Hadoop 2.2, then: a) the `org.apache.spark.deploy.yarn` package will be compiled from the `new-yarn` directory. b) Protobuf v2.5 will be used as a Spark dependency, since Hadoop 2.2 depends on it. Also, Spark will be built against a version of Akka v2.0.5 that's built against Protobuf 2.5, named `akka-2.0.5-protobuf-2.5`. The patched Akka is here: https://github.com/harveyfeng/akka/tree/2.0.5-protobuf-2.5, and was published to local Ivy during testing. There's also a new boolean environment variable, `SPARK_IS_NEW_HADOOP`, that users can manually set if their `SPARK_HADOOP_VERSION` specification does not start with `2.2`, which is how the build file tries to detect a 2.2 version. Not sure if this is necessary or done in the best way, though...
No related branches found
No related tags found
Showing
- core/pom.xml 3 additions, 7 deletionscore/pom.xml
- core/src/main/scala/org/apache/spark/SparkContext.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/SparkContext.scala
- new-yarn/pom.xml 161 additions, 0 deletionsnew-yarn/pom.xml
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 446 additions, 0 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala 94 additions, 0 deletions...apache/spark/deploy/yarn/ApplicationMasterArguments.scala
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala 519 additions, 0 deletions.../src/main/scala/org/apache/spark/deploy/yarn/Client.scala
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala 148 additions, 0 deletions.../scala/org/apache/spark/deploy/yarn/ClientArguments.scala
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientDistributedCacheManager.scala 228 additions, 0 deletions...che/spark/deploy/yarn/ClientDistributedCacheManager.scala
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala 223 additions, 0 deletions...n/scala/org/apache/spark/deploy/yarn/WorkerLauncher.scala
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/WorkerRunnable.scala 209 additions, 0 deletions...n/scala/org/apache/spark/deploy/yarn/WorkerRunnable.scala
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala 687 additions, 0 deletions.../org/apache/spark/deploy/yarn/YarnAllocationHandler.scala
- new-yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala 43 additions, 0 deletions...la/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
- new-yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientClusterScheduler.scala 47 additions, 0 deletions.../spark/scheduler/cluster/YarnClientClusterScheduler.scala
- new-yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala 109 additions, 0 deletions.../spark/scheduler/cluster/YarnClientSchedulerBackend.scala
- new-yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnClusterScheduler.scala 55 additions, 0 deletions...apache/spark/scheduler/cluster/YarnClusterScheduler.scala
- new-yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientDistributedCacheManagerSuite.scala 220 additions, 0 deletions...park/deploy/yarn/ClientDistributedCacheManagerSuite.scala
- pom.xml 52 additions, 9 deletionspom.xml
- project/SparkBuild.scala 25 additions, 9 deletionsproject/SparkBuild.scala
- streaming/pom.xml 1 addition, 8 deletionsstreaming/pom.xml
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 95 additions, 77 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
Loading
Please register or sign in to comment