Skip to content
Snippets Groups Projects
Commit 182f9b7a authored by Kun Xu's avatar Kun Xu Committed by Cheng Lian
Browse files

[SPARK-9973] [SQL] Correct in-memory columnar buffer size

The `initialSize` argument of `ColumnBuilder.initialize()` should be the
number of rows rather than bytes.  However `InMemoryColumnarTableScan`
passes in a byte size, which makes Spark SQL allocate more memory than
necessary when building in-memory columnar buffers.

Author: Kun Xu <viper_kun@163.com>

Closes #8189 from viper-kun/errorSize.
parent 1db7179f
No related branches found
No related tags found
No related merge requests found
......@@ -121,8 +121,7 @@ private[sql] case class InMemoryRelation(
def next(): CachedBatch = {
val columnBuilders = output.map { attribute =>
val columnType = ColumnType(attribute.dataType)
val initialBufferSize = columnType.defaultSize * batchSize
ColumnBuilder(attribute.dataType, initialBufferSize, attribute.name, useCompression)
ColumnBuilder(attribute.dataType, batchSize, attribute.name, useCompression)
}.toArray
var rowCount = 0
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment