Skip to content
Snippets Groups Projects
  • Marcelo Vanzin's avatar
    48978abf
    [SPARK-13576][BUILD] Don't create assembly for examples. · 48978abf
    Marcelo Vanzin authored
    As part of the goal to stop creating assemblies in Spark, this change
    modifies the mvn and sbt builds to not create an assembly for examples.
    
    Instead, dependencies are copied to the build directory (under
    target/scala-xx/jars), and in the final archive, into the "examples/jars"
    directory.
    
    To avoid having to deal too much with Windows batch files, I made examples
    run through the launcher library; the spark-submit launcher now has a
    special mode to run examples, which adds all the necessary jars to the
    spark-submit command line, and replaces the bash and batch scripts that
    were used to run examples. The scripts are now just a thin wrapper around
    spark-submit; another advantage is that now all spark-submit options are
    supported.
    
    There are a few glitches; in the mvn build, a lot of duplicated dependencies
    get copied, because they are promoted to "compile" scope due to extra
    dependencies in the examples module (such as HBase). In the sbt build,
    all dependencies are copied, because there doesn't seem to be an easy
    way to filter things.
    
    I plan to clean some of this up when the rest of the tasks are finished.
    When the main assembly is replaced with jars, we can remove duplicate jars
    from the examples directory during packaging.
    
    Tested by running SparkPi in: maven build, sbt build, dist created by
    make-distribution.sh.
    
    Finally: note that running the "assembly" target in sbt doesn't build
    the examples anymore. You need to run "package" for that.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #11452 from vanzin/SPARK-13576.
    48978abf
    History
    [SPARK-13576][BUILD] Don't create assembly for examples.
    Marcelo Vanzin authored
    As part of the goal to stop creating assemblies in Spark, this change
    modifies the mvn and sbt builds to not create an assembly for examples.
    
    Instead, dependencies are copied to the build directory (under
    target/scala-xx/jars), and in the final archive, into the "examples/jars"
    directory.
    
    To avoid having to deal too much with Windows batch files, I made examples
    run through the launcher library; the spark-submit launcher now has a
    special mode to run examples, which adds all the necessary jars to the
    spark-submit command line, and replaces the bash and batch scripts that
    were used to run examples. The scripts are now just a thin wrapper around
    spark-submit; another advantage is that now all spark-submit options are
    supported.
    
    There are a few glitches; in the mvn build, a lot of duplicated dependencies
    get copied, because they are promoted to "compile" scope due to extra
    dependencies in the examples module (such as HBase). In the sbt build,
    all dependencies are copied, because there doesn't seem to be an easy
    way to filter things.
    
    I plan to clean some of this up when the rest of the tasks are finished.
    When the main assembly is replaced with jars, we can remove duplicate jars
    from the examples directory during packaging.
    
    Tested by running SparkPi in: maven build, sbt build, dist created by
    make-distribution.sh.
    
    Finally: note that running the "assembly" target in sbt doesn't build
    the examples anymore. You need to run "package" for that.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #11452 from vanzin/SPARK-13576.