Skip to content
Snippets Groups Projects
Commit 92f8f803 authored by Shuo Xiang's avatar Shuo Xiang Committed by Joseph K. Bradley
Browse files

[SPARK-7452] [MLLIB] fix bug in topBykey and update test

the toArray function of the BoundedPriorityQueue does not necessarily preserve order. Add a counter-example as the test, which would fail the original impl.

Author: Shuo Xiang <shuoxiangpub@gmail.com>

Closes #5990 from coderxiang/topbykey-test and squashes the following commits:

98804c9 [Shuo Xiang] fix bug in topBykey and update test
parent cd1d4110
No related branches found
No related tags found
No related merge requests found
......@@ -46,7 +46,7 @@ class MLPairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD[(K, V)]) extends Se
combOp = (queue1, queue2) => {
queue1 ++= queue2
}
).mapValues(_.toArray.reverse) // This is an min-heap, so we reverse the order.
).mapValues(_.toArray.sorted(ord.reverse)) // This is an min-heap, so we reverse the order.
}
}
......
......@@ -24,13 +24,14 @@ import org.apache.spark.mllib.rdd.MLPairRDDFunctions._
class MLPairRDDFunctionsSuite extends FunSuite with MLlibTestSparkContext {
test("topByKey") {
val topMap = sc.parallelize(Array((1, 1), (1, 2), (3, 2), (3, 7), (5, 1), (3, 5)), 2)
.topByKey(2)
val topMap = sc.parallelize(Array((1, 7), (1, 3), (1, 6), (1, 1), (1, 2), (3, 2), (3, 7), (5,
1), (3, 5)), 2)
.topByKey(5)
.collectAsMap()
assert(topMap.size === 3)
assert(topMap(1) === Array(2, 1))
assert(topMap(3) === Array(7, 5))
assert(topMap(1) === Array(7, 6, 3, 2, 1))
assert(topMap(3) === Array(7, 5, 2))
assert(topMap(5) === Array(1))
}
}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment