-
- Downloads
[SPARK-9404][SPARK-9542][SQL] unsafe array data and map data
This PR adds a UnsafeArrayData, current we encode it in this way: first 4 bytes is the # elements then each 4 byte is the start offset of the element, unless it is negative, in which case the element is null. followed by the elements themselves an example: [10, 11, 12, 13, null, 14] will be encoded as: 5, 28, 32, 36, 40, -44, 44, 10, 11, 12, 13, 14 Note that, when we read a UnsafeArrayData from bytes, we can read the first 4 bytes as numElements and take the rest(first 4 bytes skipped) as value region. unsafe map data just use 2 unsafe array data, first 4 bytes is # of elements, second 4 bytes is numBytes of key array, the follows key array data and value array data. Author: Wenchen Fan <cloud0fan@outlook.com> Closes #7752 from cloud-fan/unsafe-array and squashes the following commits: 3269bd7 [Wenchen Fan] fix a bug 6445289 [Wenchen Fan] add unit tests 49adf26 [Wenchen Fan] add unsafe map 20d1039 [Wenchen Fan] add comments and unsafe converter 821b8db [Wenchen Fan] add unsafe array
Showing
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java 333 additions, 0 deletions...pache/spark/sql/catalyst/expressions/UnsafeArrayData.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeMapData.java 66 additions, 0 deletions.../apache/spark/sql/catalyst/expressions/UnsafeMapData.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeReaders.java 48 additions, 0 deletions.../apache/spark/sql/catalyst/expressions/UnsafeReaders.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java 28 additions, 6 deletions.../org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRowWriters.java 71 additions, 0 deletions...ache/spark/sql/catalyst/expressions/UnsafeRowWriters.java
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeWriters.java 208 additions, 0 deletions.../apache/spark/sql/catalyst/expressions/UnsafeWriters.java
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/FromUnsafe.scala 67 additions, 0 deletions...rg/apache/spark/sql/catalyst/expressions/FromUnsafe.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala 9 additions, 1 deletion...rg/apache/spark/sql/catalyst/expressions/Projection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala 3 additions, 1 deletion...park/sql/catalyst/expressions/codegen/CodeGenerator.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala 318 additions, 9 deletions...talyst/expressions/codegen/GenerateUnsafeProjection.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/ArrayBasedMapData.scala 12 additions, 3 deletions.../scala/org/apache/spark/sql/types/ArrayBasedMapData.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/ArrayData.scala 9 additions, 5 deletions...src/main/scala/org/apache/spark/sql/types/ArrayData.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/GenericArrayData.scala 8 additions, 2 deletions...n/scala/org/apache/spark/sql/types/GenericArrayData.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/MapData.scala 2 additions, 0 deletions...t/src/main/scala/org/apache/spark/sql/types/MapData.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/UnsafeRowConverterSuite.scala 110 additions, 4 deletions...rk/sql/catalyst/expressions/UnsafeRowConverterSuite.scala
- unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java 3 additions, 0 deletions...c/main/java/org/apache/spark/unsafe/types/UTF8String.java
Loading
Please register or sign in to comment