Skip to content
  • jerryshao's avatar
    4239a108
    [SPARK-19021][YARN] Generailize HDFSCredentialProvider to support non HDFS security filesystems · 4239a108
    jerryshao authored
    Currently Spark can only get token renewal interval from security HDFS (hdfs://), if Spark runs with other security file systems like webHDFS (webhdfs://), wasb (wasb://), ADLS, it will ignore these tokens and not get token renewal intervals from these tokens. These will make Spark unable to work with these security clusters. So instead of only checking HDFS token, we should generalize to support different DelegationTokenIdentifier.
    
    ## How was this patch tested?
    
    Manually verified in security cluster.
    
    Author: jerryshao <sshao@hortonworks.com>
    
    Closes #16432 from jerryshao/SPARK-19021.
    4239a108
    [SPARK-19021][YARN] Generailize HDFSCredentialProvider to support non HDFS security filesystems
    jerryshao authored
    Currently Spark can only get token renewal interval from security HDFS (hdfs://), if Spark runs with other security file systems like webHDFS (webhdfs://), wasb (wasb://), ADLS, it will ignore these tokens and not get token renewal intervals from these tokens. These will make Spark unable to work with these security clusters. So instead of only checking HDFS token, we should generalize to support different DelegationTokenIdentifier.
    
    ## How was this patch tested?
    
    Manually verified in security cluster.
    
    Author: jerryshao <sshao@hortonworks.com>
    
    Closes #16432 from jerryshao/SPARK-19021.
Loading