Skip to content
Snippets Groups Projects
  • WangTaoTheTonic's avatar
    a51b133d
    [SPARK-7524] [SPARK-7846] add configs for keytab and principal, pass these two... · a51b133d
    WangTaoTheTonic authored
    [SPARK-7524] [SPARK-7846] add configs for keytab and principal, pass these two configs with different way in different modes
    
    * As spark now supports long running service by updating tokens for namenode, but only accept parameters passed with "--k=v" format which is not very convinient. This patch add spark.* configs in properties file and system property.
    
    *  --principal and --keytabl options are passed to client but when we started thrift server or spark-shell these two are also passed into the Main class (org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 and org.apache.spark.repl.Main).
    In these two main class, arguments passed in will be processed with some 3rd libraries, which will lead to some error: "Invalid option: --principal" or "Unrecgnised option: --principal".
    We should pass these command args in different forms, say system properties.
    
    Author: WangTaoTheTonic <wangtao111@huawei.com>
    
    Closes #6051 from WangTaoTheTonic/SPARK-7524 and squashes the following commits:
    
    e65699a [WangTaoTheTonic] change logic to loadEnvironments
    ebd9ea0 [WangTaoTheTonic] merge master
    ecfe43a [WangTaoTheTonic] pass keytab and principal seperately in different mode
    33a7f40 [WangTaoTheTonic] expand the use of the current configs
    08bb4e8 [WangTaoTheTonic] fix wrong cite
    73afa64 [WangTaoTheTonic] add configs for keytab and principal, move originals to internal
    a51b133d
    History
    [SPARK-7524] [SPARK-7846] add configs for keytab and principal, pass these two...
    WangTaoTheTonic authored
    [SPARK-7524] [SPARK-7846] add configs for keytab and principal, pass these two configs with different way in different modes
    
    * As spark now supports long running service by updating tokens for namenode, but only accept parameters passed with "--k=v" format which is not very convinient. This patch add spark.* configs in properties file and system property.
    
    *  --principal and --keytabl options are passed to client but when we started thrift server or spark-shell these two are also passed into the Main class (org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 and org.apache.spark.repl.Main).
    In these two main class, arguments passed in will be processed with some 3rd libraries, which will lead to some error: "Invalid option: --principal" or "Unrecgnised option: --principal".
    We should pass these command args in different forms, say system properties.
    
    Author: WangTaoTheTonic <wangtao111@huawei.com>
    
    Closes #6051 from WangTaoTheTonic/SPARK-7524 and squashes the following commits:
    
    e65699a [WangTaoTheTonic] change logic to loadEnvironments
    ebd9ea0 [WangTaoTheTonic] merge master
    ecfe43a [WangTaoTheTonic] pass keytab and principal seperately in different mode
    33a7f40 [WangTaoTheTonic] expand the use of the current configs
    08bb4e8 [WangTaoTheTonic] fix wrong cite
    73afa64 [WangTaoTheTonic] add configs for keytab and principal, move originals to internal