-
- Downloads
[SPARK-3032][Shuffle] Fix key comparison integer overflow introduced sorting exception
Previous key comparison in `ExternalSorter` will get wrong sorting result or exception when key comparison overflows, details can be seen in [SPARK-3032](https://issues.apache.org/jira/browse/SPARK-3032 ). Here fix this and add a unit test to prove it. Author: jerryshao <saisai.shao@intel.com> Closes #2514 from jerryshao/SPARK-3032 and squashes the following commits: 6f3c302 [jerryshao] Improve the unit test according to comments 01911e6 [jerryshao] Change the test to show the contract violate exception 83acb38 [jerryshao] Minor changes according to comments fa2a08f [jerryshao] Fix key comparison integer overflow introduced sorting exception (cherry picked from commit dab1b0ae) Signed-off-by:Matei Zaharia <matei@databricks.com>
Showing
- core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala 1 addition, 1 deletion...ala/org/apache/spark/util/collection/ExternalSorter.scala
- core/src/test/scala/org/apache/spark/util/collection/ExternalSorterSuite.scala 55 additions, 0 deletions...rg/apache/spark/util/collection/ExternalSorterSuite.scala
Loading
Please register or sign in to comment