Skip to content
Snippets Groups Projects
  • Marcelo Vanzin's avatar
    97173893
    [SPARK-6890] [core] Fix launcher lib work with SPARK_PREPEND_CLASSES. · 97173893
    Marcelo Vanzin authored
    The fix for SPARK-6406 broke the case where sub-processes are launched
    when SPARK_PREPEND_CLASSES is set, because the code now would only add
    the launcher's build directory to the sub-process's classpath instead
    of the complete assembly.
    
    This patch fixes the problem by having the launch scripts stash the
    assembly's location in an environment variable. This is not the prettiest
    solution, but it avoids having to plumb that location all the way through
    the Worker code that launches executors. The env variable is always
    set by the launch scripts, so users cannot override it.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #5504 from vanzin/SPARK-6890 and squashes the following commits:
    
    7aec921 [Marcelo Vanzin] Fix tests.
    ff87a60 [Marcelo Vanzin] Merge branch 'master' into SPARK-6890
    31d3ce8 [Marcelo Vanzin] [SPARK-6890] [core] Fix launcher lib work with SPARK_PREPEND_CLASSES.
    97173893
    History
    [SPARK-6890] [core] Fix launcher lib work with SPARK_PREPEND_CLASSES.
    Marcelo Vanzin authored
    The fix for SPARK-6406 broke the case where sub-processes are launched
    when SPARK_PREPEND_CLASSES is set, because the code now would only add
    the launcher's build directory to the sub-process's classpath instead
    of the complete assembly.
    
    This patch fixes the problem by having the launch scripts stash the
    assembly's location in an environment variable. This is not the prettiest
    solution, but it avoids having to plumb that location all the way through
    the Worker code that launches executors. The env variable is always
    set by the launch scripts, so users cannot override it.
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #5504 from vanzin/SPARK-6890 and squashes the following commits:
    
    7aec921 [Marcelo Vanzin] Fix tests.
    ff87a60 [Marcelo Vanzin] Merge branch 'master' into SPARK-6890
    31d3ce8 [Marcelo Vanzin] [SPARK-6890] [core] Fix launcher lib work with SPARK_PREPEND_CLASSES.