-
- Downloads
[SPARK-17528][SQL] data should be copied properly before saving into InternalRow
## What changes were proposed in this pull request? For performance reasons, `UnsafeRow.getString`, `getStruct`, etc. return a "pointer" that points to a memory region of this unsafe row. This makes the unsafe projection a little dangerous, because all of its output rows share one instance. When we implement SQL operators, we should be careful to not cache the input rows because they may be produced by unsafe projection from child operator and thus its content may change overtime. However, when we updating values of InternalRow(e.g. in mutable projection and safe projection), we only copy UTF8String, we should also copy InternalRow, ArrayData and MapData. This PR fixes this, and also fixes the copy of vairous InternalRow, ArrayData and MapData implementations. ## How was this patch tested? new regression tests Author: Wenchen Fan <wenchen@databricks.com> Closes #18483 from cloud-fan/fix-copy.
Showing
- common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java 6 additions, 0 deletions...c/main/java/org/apache/spark/unsafe/types/UTF8String.java
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala 26 additions, 1 deletion...ain/scala/org/apache/spark/sql/catalyst/InternalRow.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala 1 addition, 1 deletion...cala/org/apache/spark/sql/catalyst/expressions/Cast.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SpecificInternalRow.scala 0 additions, 12 deletions.../spark/sql/catalyst/expressions/SpecificInternalRow.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala 1 addition, 1 deletion...he/spark/sql/catalyst/expressions/aggregate/collect.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala 6 additions, 0 deletions...spark/sql/catalyst/expressions/aggregate/interfaces.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala 4 additions, 2 deletions...park/sql/catalyst/expressions/codegen/CodeGenerator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateSafeProjection.scala 0 additions, 2 deletions...catalyst/expressions/codegen/GenerateSafeProjection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala 11 additions, 12 deletions...cala/org/apache/spark/sql/catalyst/expressions/rows.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala 9 additions, 1 deletion...org/apache/spark/sql/catalyst/util/GenericArrayData.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/RowTest.scala 0 additions, 4 deletions...atalyst/src/test/scala/org/apache/spark/sql/RowTest.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GeneratedProjectionSuite.scala 36 additions, 0 deletions...talyst/expressions/codegen/GeneratedProjectionSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/ComplexDataSuite.scala 107 additions, 0 deletions...org/apache/spark/sql/catalyst/util/ComplexDataSuite.scala
- sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarBatch.java 1 addition, 1 deletion.../apache/spark/sql/execution/vectorized/ColumnarBatch.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortBasedAggregationIterator.scala 2 additions, 13 deletions...ql/execution/aggregate/SortBasedAggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/GenerateColumnAccessor.scala 0 additions, 1 deletion...spark/sql/execution/columnar/GenerateColumnAccessor.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/window/AggregateProcessor.scala 2 additions, 5 deletions...pache/spark/sql/execution/window/AggregateProcessor.scala
Loading
Please register or sign in to comment