Skip to content
Snippets Groups Projects
Commit 85c6ce61 authored by jinxing's avatar jinxing Committed by Tom Graves
Browse files

[SPARK-20426] Lazy initialization of FileSegmentManagedBuffer for shuffle service. 

## What changes were proposed in this pull request?
When application contains large amount of shuffle blocks. NodeManager requires lots of memory to keep metadata(`FileSegmentManagedBuffer`) in `StreamManager`. When the number of shuffle blocks is big enough. NodeManager can run OOM. This pr proposes to do lazy initialization of `FileSegmentManagedBuffer` in shuffle service.

## How was this patch tested?

Manually test.

Author: jinxing <jinxing6042@126.com>

Closes #17744 from jinxing64/SPARK-20426.
parent 561e9cc3
No related branches found
No related tags found
Hide whitespace changes
Inline Side-by-side
Showing
with 29 additions and 20 deletions
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment