Skip to content
Snippets Groups Projects
  1. Aug 29, 2013
    • Matei Zaharia's avatar
      Update Maven build to create assemblies expected by new scripts · 666d93c2
      Matei Zaharia authored
      This includes the following changes:
      - The "assembly" package now builds in Maven by default, and creates an
        assembly containing both hadoop-client and Spark, unlike the old
        BigTop distribution assembly that skipped hadoop-client
      - There is now a bigtop-dist package to build the old BigTop assembly
      - The repl-bin package is no longer built by default since the scripts
        don't reply on it; instead it can be enabled with -Prepl-bin
      - Py4J is now included in the assembly/lib folder as a local Maven repo,
        so that the Maven package can link to it
      - run-example now adds the original Spark classpath as well because the
        Maven examples assembly lists spark-core and such as provided
      - The various Maven projects add a spark-yarn dependency correctly
      666d93c2
    • Matei Zaharia's avatar
      ab0e625d
  2. Jan 01, 2013
  3. Dec 28, 2012
    • Josh Rosen's avatar
      Simplify PySpark installation. · 665466df
      Josh Rosen authored
      - Bundle Py4J binaries, since it's hard to install
      - Uses Spark's `run` script to launch the Py4J
        gateway, inheriting the settings in spark-env.sh
      
      With these changes, (hopefully) nothing more than
      running `sbt/sbt package` will be necessary to run
      PySpark.
      665466df
Loading