Skip to content
Snippets Groups Projects
  • Patrick Wendell's avatar
    1c04652c
    SPARK-1843: Replace assemble-deps with env variable. · 1c04652c
    Patrick Wendell authored
    (This change is actually small, I moved some logic into
    compute-classpath that was previously in spark-class).
    
    Assemble deps has existed for a while to allow developers to
    run local code with new changes quickly. When I'm developing I
    typically use a simpler approach which just prepends the Spark
    classes to the classpath before the assembly jar. This is well
    defined in the JVM and the Spark classes take precedence over those
    in the assembly.
    
    This approach is portable across both builds which is the main reason I'd
    like to switch to it. It's also a bit easier to toggle on and off quickly.
    
    The way you use this is the following:
    ```
    $ ./bin/spark-shell # Use spark with the normal assembly
    $ export SPARK_PREPEND_CLASSES=true
    $ ./bin/spark-shell # Now it's using compiled classes
    $ unset SPARK_PREPEND_CLASSES
    $ ./bin/spark-shell # Back to normal
    ```
    
    Author: Patrick Wendell <pwendell@gmail.com>
    
    Closes #877 from pwendell/assemble-deps and squashes the following commits:
    
    8a11345 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into assemble-deps
    faa3168 [Patrick Wendell] Adding a warning for compatibility
    3f151a7 [Patrick Wendell] Small fix
    bbfb73c [Patrick Wendell] Review feedback
    328e9f8 [Patrick Wendell] SPARK-1843: Replace assemble-deps with env variable.
    1c04652c
    History
    SPARK-1843: Replace assemble-deps with env variable.
    Patrick Wendell authored
    (This change is actually small, I moved some logic into
    compute-classpath that was previously in spark-class).
    
    Assemble deps has existed for a while to allow developers to
    run local code with new changes quickly. When I'm developing I
    typically use a simpler approach which just prepends the Spark
    classes to the classpath before the assembly jar. This is well
    defined in the JVM and the Spark classes take precedence over those
    in the assembly.
    
    This approach is portable across both builds which is the main reason I'd
    like to switch to it. It's also a bit easier to toggle on and off quickly.
    
    The way you use this is the following:
    ```
    $ ./bin/spark-shell # Use spark with the normal assembly
    $ export SPARK_PREPEND_CLASSES=true
    $ ./bin/spark-shell # Now it's using compiled classes
    $ unset SPARK_PREPEND_CLASSES
    $ ./bin/spark-shell # Back to normal
    ```
    
    Author: Patrick Wendell <pwendell@gmail.com>
    
    Closes #877 from pwendell/assemble-deps and squashes the following commits:
    
    8a11345 [Patrick Wendell] Merge remote-tracking branch 'apache/master' into assemble-deps
    faa3168 [Patrick Wendell] Adding a warning for compatibility
    3f151a7 [Patrick Wendell] Small fix
    bbfb73c [Patrick Wendell] Review feedback
    328e9f8 [Patrick Wendell] SPARK-1843: Replace assemble-deps with env variable.