-
- Downloads
[SPARK-18003][SPARK CORE] Fix bug of RDD zipWithIndex & zipWithUniqueId index value overflowing
## What changes were proposed in this pull request? - Fix bug of RDD `zipWithIndex` generating wrong result when one partition contains more than 2147483647 records. - Fix bug of RDD `zipWithUniqueId` generating wrong result when one partition contains more than 2147483647 records. ## How was this patch tested? test added. Author: WeichenXu <WeichenXu123@outlook.com> Closes #15550 from WeichenXu123/fix_rdd_zipWithIndex_overflow.
Showing
- core/src/main/scala/org/apache/spark/rdd/RDD.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/rdd/RDD.scala
- core/src/main/scala/org/apache/spark/rdd/ZippedWithIndexRDD.scala 2 additions, 3 deletions.../main/scala/org/apache/spark/rdd/ZippedWithIndexRDD.scala
- core/src/main/scala/org/apache/spark/util/Utils.scala 15 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/Utils.scala
- core/src/test/scala/org/apache/spark/util/UtilsSuite.scala 7 additions, 0 deletionscore/src/test/scala/org/apache/spark/util/UtilsSuite.scala
Loading
Please register or sign in to comment