Skip to content
Snippets Groups Projects
  • WangTaoTheTonic's avatar
    b65bad65
    [SPARK-3591][YARN]fire and forget for YARN cluster mode · b65bad65
    WangTaoTheTonic authored
    https://issues.apache.org/jira/browse/SPARK-3591
    
    The output after this patch:
    >doggie153:/opt/oss/spark-1.3.0-bin-hadoop2.4/bin # ./spark-submit  --class org.apache.spark.examples.SparkPi --master yarn-cluster ../lib/spark-examples*.jar
    15/03/31 21:15:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    15/03/31 21:15:25 INFO RMProxy: Connecting to ResourceManager at doggie153/10.177.112.153:8032
    15/03/31 21:15:25 INFO Client: Requesting a new application from cluster with 4 NodeManagers
    15/03/31 21:15:25 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
    15/03/31 21:15:25 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
    15/03/31 21:15:25 INFO Client: Setting up container launch context for our AM
    15/03/31 21:15:25 INFO Client: Preparing resources for our AM container
    15/03/31 21:15:26 INFO Client: Uploading resource file:/opt/oss/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.4.0-SNAPSHOT-hadoop2.4.1.jar -> hdfs://doggie153:9000/user/root/.sparkStaging/application_1427257505534_0016/spark-assembly-1.4.0-SNAPSHOT-hadoop2.4.1.jar
    15/03/31 21:15:27 INFO Client: Uploading resource file:/opt/oss/spark-1.3.0-bin-hadoop2.4/lib/spark-examples-1.3.0-hadoop2.4.0.jar -> hdfs://doggie153:9000/user/root/.sparkStaging/application_1427257505534_0016/spark-examples-1.3.0-hadoop2.4.0.jar
    15/03/31 21:15:28 INFO Client: Setting up the launch environment for our AM container
    15/03/31 21:15:28 INFO SecurityManager: Changing view acls to: root
    15/03/31 21:15:28 INFO SecurityManager: Changing modify acls to: root
    15/03/31 21:15:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
    15/03/31 21:15:28 INFO Client: Submitting application 16 to ResourceManager
    15/03/31 21:15:28 INFO YarnClientImpl: Submitted application application_1427257505534_0016
    15/03/31 21:15:28 INFO Client: ... waiting before polling ResourceManager for application state
    15/03/31 21:15:33 INFO Client: ... polling ResourceManager for application state
    15/03/31 21:15:33 INFO Client: Application report for application_1427257505534_0016 (state: RUNNING)
    15/03/31 21:15:33 INFO Client:
             client token: N/A
             diagnostics: N/A
             ApplicationMaster host: doggie157
             ApplicationMaster RPC port: 0
             queue: default
             start time: 1427807728307
             final status: UNDEFINED
             tracking URL: http://doggie153:8088/proxy/application_1427257505534_0016/
             user: root
    
    /cc andrewor14
    
    Author: WangTaoTheTonic <wangtao111@huawei.com>
    
    Closes #5297 from WangTaoTheTonic/SPARK-3591 and squashes the following commits:
    
    c76d232 [WangTaoTheTonic] wrap lines
    16c90a8 [WangTaoTheTonic] move up lines to avoid duplicate
    fea390d [WangTaoTheTonic] log failed/killed report, style and comment
    be1cc2e [WangTaoTheTonic] reword
    f0bc54f [WangTaoTheTonic] minor: expose appid in excepiton messages
    ba9b22b [WangTaoTheTonic] wrong config name
    e1a4013 [WangTaoTheTonic] revert to the old version and do some robust
    19706c0 [WangTaoTheTonic] add a config to control whether to forget
    0cbdce8 [WangTaoTheTonic] fire and forget for YARN cluster mode
    b65bad65
    History
    [SPARK-3591][YARN]fire and forget for YARN cluster mode
    WangTaoTheTonic authored
    https://issues.apache.org/jira/browse/SPARK-3591
    
    The output after this patch:
    >doggie153:/opt/oss/spark-1.3.0-bin-hadoop2.4/bin # ./spark-submit  --class org.apache.spark.examples.SparkPi --master yarn-cluster ../lib/spark-examples*.jar
    15/03/31 21:15:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    15/03/31 21:15:25 INFO RMProxy: Connecting to ResourceManager at doggie153/10.177.112.153:8032
    15/03/31 21:15:25 INFO Client: Requesting a new application from cluster with 4 NodeManagers
    15/03/31 21:15:25 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
    15/03/31 21:15:25 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
    15/03/31 21:15:25 INFO Client: Setting up container launch context for our AM
    15/03/31 21:15:25 INFO Client: Preparing resources for our AM container
    15/03/31 21:15:26 INFO Client: Uploading resource file:/opt/oss/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.4.0-SNAPSHOT-hadoop2.4.1.jar -> hdfs://doggie153:9000/user/root/.sparkStaging/application_1427257505534_0016/spark-assembly-1.4.0-SNAPSHOT-hadoop2.4.1.jar
    15/03/31 21:15:27 INFO Client: Uploading resource file:/opt/oss/spark-1.3.0-bin-hadoop2.4/lib/spark-examples-1.3.0-hadoop2.4.0.jar -> hdfs://doggie153:9000/user/root/.sparkStaging/application_1427257505534_0016/spark-examples-1.3.0-hadoop2.4.0.jar
    15/03/31 21:15:28 INFO Client: Setting up the launch environment for our AM container
    15/03/31 21:15:28 INFO SecurityManager: Changing view acls to: root
    15/03/31 21:15:28 INFO SecurityManager: Changing modify acls to: root
    15/03/31 21:15:28 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
    15/03/31 21:15:28 INFO Client: Submitting application 16 to ResourceManager
    15/03/31 21:15:28 INFO YarnClientImpl: Submitted application application_1427257505534_0016
    15/03/31 21:15:28 INFO Client: ... waiting before polling ResourceManager for application state
    15/03/31 21:15:33 INFO Client: ... polling ResourceManager for application state
    15/03/31 21:15:33 INFO Client: Application report for application_1427257505534_0016 (state: RUNNING)
    15/03/31 21:15:33 INFO Client:
             client token: N/A
             diagnostics: N/A
             ApplicationMaster host: doggie157
             ApplicationMaster RPC port: 0
             queue: default
             start time: 1427807728307
             final status: UNDEFINED
             tracking URL: http://doggie153:8088/proxy/application_1427257505534_0016/
             user: root
    
    /cc andrewor14
    
    Author: WangTaoTheTonic <wangtao111@huawei.com>
    
    Closes #5297 from WangTaoTheTonic/SPARK-3591 and squashes the following commits:
    
    c76d232 [WangTaoTheTonic] wrap lines
    16c90a8 [WangTaoTheTonic] move up lines to avoid duplicate
    fea390d [WangTaoTheTonic] log failed/killed report, style and comment
    be1cc2e [WangTaoTheTonic] reword
    f0bc54f [WangTaoTheTonic] minor: expose appid in excepiton messages
    ba9b22b [WangTaoTheTonic] wrong config name
    e1a4013 [WangTaoTheTonic] revert to the old version and do some robust
    19706c0 [WangTaoTheTonic] add a config to control whether to forget
    0cbdce8 [WangTaoTheTonic] fire and forget for YARN cluster mode