-
- Downloads
[SPARK-21090][CORE] Optimize the unified memory manager code
## What changes were proposed in this pull request? 1.In `acquireStorageMemory`, when the Memory Mode is OFF_HEAP ,the `maxOffHeapMemory` should be modified to `maxOffHeapStorageMemory`. after this PR,it will same as ON_HEAP Memory Mode. Because when acquire memory is between `maxOffHeapStorageMemory` and `maxOffHeapMemory`,it will fail surely, so if acquire memory is greater than `maxOffHeapStorageMemory`(not greater than `maxOffHeapMemory`),we should fail fast. 2. Borrow memory from execution, `numBytes` modified to `numBytes - storagePool.memoryFree` will be more reasonable. Because we just acquire `(numBytes - storagePool.memoryFree)`, unnecessary borrowed `numBytes` from execution ## How was this patch tested? added unit test case Author: liuxian <liu.xian3@zte.com.cn> Closes #18296 from 10110346/wip-lx-0614.
Showing
- core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala 3 additions, 2 deletions.../scala/org/apache/spark/memory/UnifiedMemoryManager.scala
- core/src/test/scala/org/apache/spark/memory/MemoryManagerSuite.scala 1 addition, 1 deletion...st/scala/org/apache/spark/memory/MemoryManagerSuite.scala
- core/src/test/scala/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala 32 additions, 0 deletions...a/org/apache/spark/memory/UnifiedMemoryManagerSuite.scala
Loading
Please register or sign in to comment