Skip to content
Snippets Groups Projects
Commit 2bd81263 authored by Zhang, Liye's avatar Zhang, Liye Committed by Reynold Xin
Browse files

[SPARK-1777 (partial)] bugfix: make size of requested memory correctly

Author: Zhang, Liye <liye.zhang@intel.com>

Closes #1892 from liyezhang556520/lazy_memory_request and squashes the following commits:

335ab61 [Zhang, Liye] [SPARK-1777 (partial)] bugfix: make size of requested memory correctly
parent 246cb3f1
No related branches found
No related tags found
No related merge requests found
......@@ -238,7 +238,7 @@ private[spark] class MemoryStore(blockManager: BlockManager, maxMemory: Long)
// If our vector's size has exceeded the threshold, request more memory
val currentSize = vector.estimateSize()
if (currentSize >= memoryThreshold) {
val amountToRequest = (currentSize * (memoryGrowthFactor - 1)).toLong
val amountToRequest = (currentSize * memoryGrowthFactor - memoryThreshold).toLong
// Hold the accounting lock, in case another thread concurrently puts a block that
// takes up the unrolling space we just ensured here
accountingLock.synchronized {
......@@ -254,7 +254,7 @@ private[spark] class MemoryStore(blockManager: BlockManager, maxMemory: Long)
}
}
// New threshold is currentSize * memoryGrowthFactor
memoryThreshold = currentSize + amountToRequest
memoryThreshold += amountToRequest
}
}
elementsUnrolled += 1
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment