Skip to content
Snippets Groups Projects
  • Andrew Or's avatar
    2ffd1eaf
    [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc. · 2ffd1eaf
    Andrew Or authored
    YARN
    - SparkPi was updated to not take in master as an argument; we should update the docs to reflect that.
    - The default YARN build guide should be in maven, not sbt.
    - This PR also adds a paragraph on steps to debug a YARN application.
    
    Standalone
    - Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`.
    - The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made.
    
    In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #701 from andrewor14/yarn-docs and squashes the following commits:
    
    e2c2312 [Andrew Or] Merge in changes in #752 (SPARK-1814)
    25cfe7b [Andrew Or] Merge in the warning from SPARK-1753
    a8c39c5 [Andrew Or] Minor changes
    336bbd9 [Andrew Or] Tabs -> spaces
    4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html
    3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions
    5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc.
    c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    381fe32 [Andrew Or] Update docs for standalone mode
    757c184 [Andrew Or] Add a note about the requirements for the debugging trick
    f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    924f04c [Andrew Or] Revert addition of --deploy-mode
    d5fe17b [Andrew Or] Update the YARN docs
    2ffd1eaf
    History
    [SPARK-1753 / 1773 / 1814] Update outdated docs for spark-submit, YARN, standalone etc.
    Andrew Or authored
    YARN
    - SparkPi was updated to not take in master as an argument; we should update the docs to reflect that.
    - The default YARN build guide should be in maven, not sbt.
    - This PR also adds a paragraph on steps to debug a YARN application.
    
    Standalone
    - Emphasize spark-submit more. Right now it's one small paragraph preceding the legacy way of launching through `org.apache.spark.deploy.Client`.
    - The way we set configurations / environment variables according to the old docs is outdated. This needs to reflect changes introduced by the Spark configuration changes we made.
    
    In general, this PR also adds a little more documentation on the new spark-shell, spark-submit, spark-defaults.conf etc here and there.
    
    Author: Andrew Or <andrewor14@gmail.com>
    
    Closes #701 from andrewor14/yarn-docs and squashes the following commits:
    
    e2c2312 [Andrew Or] Merge in changes in #752 (SPARK-1814)
    25cfe7b [Andrew Or] Merge in the warning from SPARK-1753
    a8c39c5 [Andrew Or] Minor changes
    336bbd9 [Andrew Or] Tabs -> spaces
    4d9d8f7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    041017a [Andrew Or] Abstract Spark submit documentation to cluster-overview.html
    3cc0649 [Andrew Or] Detail how to set configurations + remove legacy instructions
    5b7140a [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    85a51fc [Andrew Or] Update run-example, spark-shell, configuration etc.
    c10e8c7 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    381fe32 [Andrew Or] Update docs for standalone mode
    757c184 [Andrew Or] Add a note about the requirements for the debugging trick
    f8ca990 [Andrew Or] Merge branch 'master' of github.com:apache/spark into yarn-docs
    924f04c [Andrew Or] Revert addition of --deploy-mode
    d5fe17b [Andrew Or] Update the YARN docs