-
- Downloads
Fix ClassCastException in JavaPairRDD.collectAsMap() (SPARK-1040)
This fixes an issue where collectAsMap() could fail when called on a JavaPairRDD that was derived by transforming a non-JavaPairRDD. The root problem was that we were creating the JavaPairRDD's ClassTag by casting a ClassTag[AnyRef] to a ClassTag[Tuple2[K2, V2]]. To fix this, I cast a ClassTag[Tuple2[_, _]] instead, since this actually produces a ClassTag of the appropriate type because ClassTags don't capture type parameters: scala> implicitly[ClassTag[Tuple2[_, _]]] == implicitly[ClassTag[Tuple2[Int, Int]]] res8: Boolean = true scala> implicitly[ClassTag[AnyRef]].asInstanceOf[ClassTag[Tuple2[Int, Int]]] == implicitly[ClassTag[Tuple2[Int, Int]]] res9: Boolean = false
Showing
- core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala 2 additions, 2 deletions...rc/main/scala/org/apache/spark/api/java/JavaRDDLike.scala
- core/src/test/scala/org/apache/spark/JavaAPISuite.java 17 additions, 0 deletionscore/src/test/scala/org/apache/spark/JavaAPISuite.java
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaDStreamLike.scala 2 additions, 2 deletions...org/apache/spark/streaming/api/java/JavaDStreamLike.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala 1 addition, 1 deletion...org/apache/spark/streaming/api/java/JavaPairDStream.scala
Loading
Please register or sign in to comment