Skip to content
Snippets Groups Projects
Commit d3b7671c authored by Michael Armbrust's avatar Michael Armbrust Committed by Reynold Xin
Browse files

[SQL] Improve Speed of InsertIntoHiveTable

Author: Michael Armbrust <michael@databricks.com>

Closes #1130 from marmbrus/noFunctional and squashes the following commits:

ccdb68c [Michael Armbrust] Remove functional programming and Array allocations from fast path in InsertIntoHiveTable.
parent 278ec8a2
No related branches found
No related tags found
No related merge requests found
......@@ -371,12 +371,18 @@ case class InsertIntoHiveTable(
ObjectInspectorCopyOption.JAVA)
.asInstanceOf[StructObjectInspector]
val fieldOIs = standardOI.getAllStructFieldRefs.map(_.getFieldObjectInspector).toArray
val outputData = new Array[Any](fieldOIs.length)
iter.map { row =>
// Casts Strings to HiveVarchars when necessary.
val fieldOIs = standardOI.getAllStructFieldRefs.map(_.getFieldObjectInspector)
val mappedRow = row.zip(fieldOIs).map(wrap)
var i = 0
while (i < row.length) {
// Casts Strings to HiveVarchars when necessary.
outputData(i) = wrap(row(i), fieldOIs(i))
i += 1
}
serializer.serialize(mappedRow.toArray, standardOI)
serializer.serialize(outputData, standardOI)
}
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment