Skip to content
Snippets Groups Projects
  1. Sep 23, 2013
  2. Sep 22, 2013
  3. Sep 11, 2013
  4. Aug 30, 2013
  5. Aug 29, 2013
    • Matei Zaharia's avatar
      Update Maven build to create assemblies expected by new scripts · 666d93c2
      Matei Zaharia authored
      This includes the following changes:
      - The "assembly" package now builds in Maven by default, and creates an
        assembly containing both hadoop-client and Spark, unlike the old
        BigTop distribution assembly that skipped hadoop-client
      - There is now a bigtop-dist package to build the old BigTop assembly
      - The repl-bin package is no longer built by default since the scripts
        don't reply on it; instead it can be enabled with -Prepl-bin
      - Py4J is now included in the assembly/lib folder as a local Maven repo,
        so that the Maven package can link to it
      - run-example now adds the original Spark classpath as well because the
        Maven examples assembly lists spark-core and such as provided
      - The various Maven projects add a spark-yarn dependency correctly
      666d93c2
    • Matei Zaharia's avatar
      Change build and run instructions to use assemblies · 53cd50c0
      Matei Zaharia authored
      This commit makes Spark invocation saner by using an assembly JAR to
      find all of Spark's dependencies instead of adding all the JARs in
      lib_managed. It also packages the examples into an assembly and uses
      that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script
      with two better-named scripts: "run-examples" for examples, and
      "spark-class" for Spark internal classes (e.g. REPL, master, etc). This
      is also designed to minimize the confusion people have in trying to use
      "run" to run their own classes; it's not meant to do that, but now at
      least if they look at it, they can modify run-examples to do a decent
      job for them.
      
      As part of this, Bagel's examples are also now properly moved to the
      examples package instead of bagel.
      53cd50c0
Loading