-
- Downloads
[SPARK-8309] [CORE] Support for more than 12M items in OpenHashMap
The problem occurs because the position mask `0xEFFFFFF` is incorrect. It has zero 25th bit, so when capacity grows beyond 2^24, `OpenHashMap` calculates incorrect index of value in `_values` array. I've also added a size check in `rehash()`, so that it fails instead of reporting invalid item indices. Author: Vyacheslav Baranov <slavik.baranov@gmail.com> Closes #6763 from SlavikBaranov/SPARK-8309 and squashes the following commits: 8557445 [Vyacheslav Baranov] Resolved review comments 4d5b954 [Vyacheslav Baranov] Resolved review comments eaf1e68 [Vyacheslav Baranov] Fixed failing test f9284fd [Vyacheslav Baranov] Resolved review comments 3920656 [Vyacheslav Baranov] SPARK-8309: Support for more than 12M items in OpenHashMap
Showing
- core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala 7 additions, 3 deletions.../scala/org/apache/spark/util/collection/OpenHashSet.scala
- core/src/test/scala/org/apache/spark/util/collection/OpenHashMapSuite.scala 11 additions, 1 deletion...a/org/apache/spark/util/collection/OpenHashMapSuite.scala
- core/src/test/scala/org/apache/spark/util/collection/PrimitiveKeyOpenHashMapSuite.scala 1 addition, 1 deletion.../spark/util/collection/PrimitiveKeyOpenHashMapSuite.scala
Loading
Please register or sign in to comment