Skip to content
  • Marcelo Vanzin's avatar
    37bf76a2
    [SPARK-8302] Support heterogeneous cluster install paths on YARN. · 37bf76a2
    Marcelo Vanzin authored
    Some users have Hadoop installations on different paths across
    their cluster. Currently, that makes it hard to set up some
    configuration in Spark since that requires hardcoding paths to
    jar files or native libraries, which wouldn't work on such a cluster.
    
    This change introduces a couple of YARN-specific configurations
    that instruct the backend to replace certain paths when launching
    remote processes. That way, if the configuration says the Spark
    jar is in "/spark/spark.jar", and also says that "/spark" should be
    replaced with "{{SPARK_INSTALL_DIR}}", YARN will start containers
    in the NMs with "{{SPARK_INSTALL_DIR}}/spark.jar" as the location
    of the jar.
    
    Coupled with YARN's environment whitelist (which allows certain
    env variables to be exposed to containers), this allows users to
    support such heterogeneous environments, as long as a single
    replacement is enough. (Otherwise, this feature would need to be
    extended to support multiple path replacements.)
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #6752 from vanzin/SPARK-8302 and squashes the following commits:
    
    4bff8d4 [Marcelo Vanzin] Add docs, rename configs.
    0aa2a02 [Marcelo Vanzin] Only do replacement for paths that need it.
    2e9cc9d [Marcelo Vanzin] Style.
    a5e1f68 [Marcelo Vanzin] [SPARK-8302] Support heterogeneous cluster install paths on YARN.
    37bf76a2
    [SPARK-8302] Support heterogeneous cluster install paths on YARN.
    Marcelo Vanzin authored
    Some users have Hadoop installations on different paths across
    their cluster. Currently, that makes it hard to set up some
    configuration in Spark since that requires hardcoding paths to
    jar files or native libraries, which wouldn't work on such a cluster.
    
    This change introduces a couple of YARN-specific configurations
    that instruct the backend to replace certain paths when launching
    remote processes. That way, if the configuration says the Spark
    jar is in "/spark/spark.jar", and also says that "/spark" should be
    replaced with "{{SPARK_INSTALL_DIR}}", YARN will start containers
    in the NMs with "{{SPARK_INSTALL_DIR}}/spark.jar" as the location
    of the jar.
    
    Coupled with YARN's environment whitelist (which allows certain
    env variables to be exposed to containers), this allows users to
    support such heterogeneous environments, as long as a single
    replacement is enough. (Otherwise, this feature would need to be
    extended to support multiple path replacements.)
    
    Author: Marcelo Vanzin <vanzin@cloudera.com>
    
    Closes #6752 from vanzin/SPARK-8302 and squashes the following commits:
    
    4bff8d4 [Marcelo Vanzin] Add docs, rename configs.
    0aa2a02 [Marcelo Vanzin] Only do replacement for paths that need it.
    2e9cc9d [Marcelo Vanzin] Style.
    a5e1f68 [Marcelo Vanzin] [SPARK-8302] Support heterogeneous cluster install paths on YARN.
Loading