Skip to content
Snippets Groups Projects
  1. Jan 06, 2014
  2. Jan 03, 2014
  3. Jan 02, 2014
  4. Sep 29, 2013
  5. Sep 23, 2013
  6. Sep 22, 2013
  7. Sep 15, 2013
  8. Sep 11, 2013
  9. Aug 30, 2013
  10. Aug 29, 2013
    • Matei Zaharia's avatar
      Update Maven build to create assemblies expected by new scripts · 666d93c2
      Matei Zaharia authored
      This includes the following changes:
      - The "assembly" package now builds in Maven by default, and creates an
        assembly containing both hadoop-client and Spark, unlike the old
        BigTop distribution assembly that skipped hadoop-client
      - There is now a bigtop-dist package to build the old BigTop assembly
      - The repl-bin package is no longer built by default since the scripts
        don't reply on it; instead it can be enabled with -Prepl-bin
      - Py4J is now included in the assembly/lib folder as a local Maven repo,
        so that the Maven package can link to it
      - run-example now adds the original Spark classpath as well because the
        Maven examples assembly lists spark-core and such as provided
      - The various Maven projects add a spark-yarn dependency correctly
      666d93c2
    • Matei Zaharia's avatar
      Change build and run instructions to use assemblies · 53cd50c0
      Matei Zaharia authored
      This commit makes Spark invocation saner by using an assembly JAR to
      find all of Spark's dependencies instead of adding all the JARs in
      lib_managed. It also packages the examples into an assembly and uses
      that as SPARK_EXAMPLES_JAR. Finally, it replaces the old "run" script
      with two better-named scripts: "run-examples" for examples, and
      "spark-class" for Spark internal classes (e.g. REPL, master, etc). This
      is also designed to minimize the confusion people have in trying to use
      "run" to run their own classes; it's not meant to do that, but now at
      least if they look at it, they can modify run-examples to do a decent
      job for them.
      
      As part of this, Bagel's examples are also now properly moved to the
      examples package instead of bagel.
      53cd50c0
Loading