-
- Downloads
[SPARK-21411][YARN] Lazily create FS within kerberized UGI to avoid token acquiring failure
## What changes were proposed in this pull request? In the current `YARNHadoopDelegationTokenManager`, `FileSystem` to which to get tokens are created out of KDC logged UGI, using these `FileSystem` to get new tokens will lead to exception. The main thing is that Spark code trying to get new tokens from the FS created with token auth-ed UGI, but Hadoop can only grant new tokens in kerberized UGI. To fix this issue, we should lazily create these FileSystem within KDC logged UGI. ## How was this patch tested? Manual verification in secure cluster. CC vanzin mgummelt please help to review, thanks! Author: jerryshao <sshao@hortonworks.com> Closes #18633 from jerryshao/SPARK-21411.
Showing
- core/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala 1 addition, 1 deletion.../spark/deploy/security/HadoopDelegationTokenManager.scala
- core/src/main/scala/org/apache/spark/deploy/security/HadoopFSDelegationTokenProvider.scala 4 additions, 3 deletions...ark/deploy/security/HadoopFSDelegationTokenProvider.scala
- core/src/test/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManagerSuite.scala 4 additions, 4 deletions...k/deploy/security/HadoopDelegationTokenManagerSuite.scala
- resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 1 addition, 1 deletion...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
- resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala 1 addition, 1 deletion.../src/main/scala/org/apache/spark/deploy/yarn/Client.scala
- resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala 1 addition, 1 deletion...la/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
- resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/YARNHadoopDelegationTokenManager.scala 1 addition, 1 deletion...ploy/yarn/security/YARNHadoopDelegationTokenManager.scala
- resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/security/YARNHadoopDelegationTokenManagerSuite.scala 1 addition, 1 deletion...yarn/security/YARNHadoopDelegationTokenManagerSuite.scala
Loading
Please register or sign in to comment