-
- Downloads
Merge pull request #435 from tdas/filestream-fix
Fixed the flaky tests by making SparkConf not serializable SparkConf was being serialized with CoGroupedRDD and Aggregator, which somehow caused OptionalJavaException while being deserialized as part of a ShuffleMapTask. SparkConf should not even be serializable (according to conversation with Matei). This change fixes that. @mateiz @pwendell
No related branches found
No related tags found
Showing
- core/src/main/scala/org/apache/spark/Aggregator.scala 1 addition, 2 deletionscore/src/main/scala/org/apache/spark/Aggregator.scala
- core/src/main/scala/org/apache/spark/SparkConf.scala 2 additions, 1 deletioncore/src/main/scala/org/apache/spark/SparkConf.scala
- core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala 1 addition, 2 deletionscore/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala
- external/flume/src/test/resources/log4j.properties 1 addition, 1 deletionexternal/flume/src/test/resources/log4j.properties
- external/kafka/src/test/resources/log4j.properties 1 addition, 1 deletionexternal/kafka/src/test/resources/log4j.properties
- external/kafka/src/test/scala/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala 1 addition, 0 deletions...a/org/apache/spark/streaming/kafka/KafkaStreamSuite.scala
- external/mqtt/src/test/resources/log4j.properties 1 addition, 1 deletionexternal/mqtt/src/test/resources/log4j.properties
- external/mqtt/src/test/scala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala 1 addition, 0 deletions...ala/org/apache/spark/streaming/mqtt/MQTTStreamSuite.scala
- external/twitter/src/test/resources/log4j.properties 1 addition, 1 deletionexternal/twitter/src/test/resources/log4j.properties
- external/twitter/src/test/scala/org/apache/spark/streaming/twitter/TwitterStreamSuite.scala 1 addition, 0 deletions...g/apache/spark/streaming/twitter/TwitterStreamSuite.scala
- external/zeromq/src/test/resources/log4j.properties 1 addition, 1 deletionexternal/zeromq/src/test/resources/log4j.properties
- external/zeromq/src/test/scala/org/apache/spark/streaming/zeromq/ZeroMQStreamSuite.scala 1 addition, 0 deletions...org/apache/spark/streaming/zeromq/ZeroMQStreamSuite.scala
- streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala 6 additions, 4 deletions...rc/main/scala/org/apache/spark/streaming/Checkpoint.scala
- streaming/src/main/scala/org/apache/spark/streaming/DStreamGraph.scala 2 additions, 0 deletions.../main/scala/org/apache/spark/streaming/DStreamGraph.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/DStreamCheckpointData.scala 25 additions, 1 deletion...pache/spark/streaming/dstream/DStreamCheckpointData.scala
- streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala 20 additions, 8 deletions...st/scala/org/apache/spark/streaming/CheckpointSuite.scala
Loading
Please register or sign in to comment