Skip to content
Snippets Groups Projects
Unverified Commit 07fcbea5 authored by Liang-Chi Hsieh's avatar Liang-Chi Hsieh Committed by Sean Owen
Browse files

[SPARK-18800][SQL] Correct the assert in UnsafeKVExternalSorter which ensures array size

## What changes were proposed in this pull request?

`UnsafeKVExternalSorter` uses `UnsafeInMemorySorter` to sort the records of `BytesToBytesMap` if it is given a map.

Currently we use the number of keys in `BytesToBytesMap` to determine if the array used for sort is enough or not. We has an assert that ensures the size of the array is enough: `map.numKeys() <= map.getArray().size() / 2`.

However, each record in the map takes two entries in the array, one is record pointer, another is key prefix. So the correct assert should be `map.numKeys() * 2 <= map.getArray().size() / 2`.

## How was this patch tested?

N/A

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: Liang-Chi Hsieh <viirya@gmail.com>

Closes #16232 from viirya/SPARK-18800-fix-UnsafeKVExternalSorter.
parent 3cff8161
No related branches found
No related tags found
No related merge requests found
...@@ -97,7 +97,9 @@ public final class UnsafeKVExternalSorter { ...@@ -97,7 +97,9 @@ public final class UnsafeKVExternalSorter {
canUseRadixSort); canUseRadixSort);
} else { } else {
// The array will be used to do in-place sort, which require half of the space to be empty. // The array will be used to do in-place sort, which require half of the space to be empty.
assert(map.numKeys() <= map.getArray().size() / 2); // Note: each record in the map takes two entries in the array, one is record pointer,
// another is the key prefix.
assert(map.numKeys() * 2 <= map.getArray().size() / 2);
// During spilling, the array in map will not be used, so we can borrow that and use it // During spilling, the array in map will not be used, so we can borrow that and use it
// as the underlying array for in-memory sorter (it's always large enough). // as the underlying array for in-memory sorter (it's always large enough).
// Since we will not grow the array, it's fine to pass `null` as consumer. // Since we will not grow the array, it's fine to pass `null` as consumer.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment