Skip to content
Snippets Groups Projects
Commit cde64add authored by jerryshao's avatar jerryshao Committed by Marcelo Vanzin
Browse files

[SPARK-21411][YARN] Lazily create FS within kerberized UGI to avoid token acquiring failure

## What changes were proposed in this pull request?

In the current `YARNHadoopDelegationTokenManager`, `FileSystem` to which to get tokens are created out of KDC logged UGI, using these `FileSystem` to get new tokens will lead to exception. The main thing is that Spark code trying to get new tokens from the FS created with token auth-ed UGI, but Hadoop can only grant new tokens in kerberized UGI. To fix this issue, we should lazily create these FileSystem within KDC logged UGI.

## How was this patch tested?

Manual verification in secure cluster.

CC vanzin mgummelt please help to review, thanks!

Author: jerryshao <sshao@hortonworks.com>

Closes #18633 from jerryshao/SPARK-21411.
parent d3f4a211
No related branches found
No related tags found
No related merge requests found
Showing
with 14 additions and 13 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment