-
- Downloads
[SPARK-9517][SQL] BytesToBytesMap should encode data the same way as UnsafeExternalSorter
BytesToBytesMap current encodes key/value data in the following format: ``` 8B key length, key data, 8B value length, value data ``` UnsafeExternalSorter, on the other hand, encodes data this way: ``` 4B record length, data ``` As a result, we cannot pass records encoded by BytesToBytesMap directly into UnsafeExternalSorter for sorting. However, if we rearrange data slightly, we can then pass the key/value records directly into UnsafeExternalSorter: ``` 4B key+value length, 4B key length, key data, value data ``` Author: Reynold Xin <rxin@databricks.com> Closes #7845 from rxin/kvsort-rebase and squashes the following commits: 5716b59 [Reynold Xin] Fixed test. 2e62ccb [Reynold Xin] Updated BytesToBytesMap's data encoding to put the key first. a51b641 [Reynold Xin] Added a KV sorter interface.
Showing
- core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java 31 additions, 27 deletions...ain/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 15 additions, 0 deletions...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java 3 additions, 3 deletions...apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java
- sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeKeyValueSorter.java 30 additions, 0 deletions.../org/apache/spark/sql/execution/UnsafeKeyValueSorter.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMap.java 36 additions, 37 deletions...e/spark/sql/execution/UnsafeFixedWidthAggregationMap.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/GeneratedAggregate.scala 16 additions, 11 deletions...a/org/apache/spark/sql/execution/GeneratedAggregate.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala 15 additions, 12 deletions...k/sql/execution/UnsafeFixedWidthAggregationMapSuite.scala
- unsafe/src/main/java/org/apache/spark/unsafe/KVIterator.java 29 additions, 0 deletionsunsafe/src/main/java/org/apache/spark/unsafe/KVIterator.java
Loading
Please register or sign in to comment