-
- Downloads
Merge pull request #289 from tdas/filestream-fix
Bug fixes for file input stream and checkpointing - Fixed bugs in the file input stream that led the stream to fail due to transient HDFS errors (listing files when a background thread it deleting fails caused errors, etc.) - Updated Spark's CheckpointRDD and Streaming's CheckpointWriter to use SparkContext.hadoopConfiguration, to allow checkpoints to be written to any HDFS compatible store requiring special configuration. - Changed the API of SparkContext.setCheckpointDir() - eliminated the unnecessary 'useExisting' parameter. Now SparkContext will always create a unique subdirectory within the user specified checkpoint directory. This is to ensure that previous checkpoint files are not accidentally overwritten. - Fixed bug where setting checkpoint directory as a relative local path caused the checkpointing to fail.
No related branches found
No related tags found
Showing
- core/src/main/scala/org/apache/spark/SparkContext.scala 8 additions, 15 deletionscore/src/main/scala/org/apache/spark/SparkContext.scala
- core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala 1 addition, 14 deletions...in/scala/org/apache/spark/api/java/JavaSparkContext.scala
- core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala 22 additions, 10 deletionscore/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala
- core/src/main/scala/org/apache/spark/rdd/RDDCheckpointData.scala 11 additions, 4 deletions...c/main/scala/org/apache/spark/rdd/RDDCheckpointData.scala
- core/src/test/scala/org/apache/spark/JavaAPISuite.java 2 additions, 2 deletionscore/src/test/scala/org/apache/spark/JavaAPISuite.java
- python/pyspark/context.py 2 additions, 7 deletionspython/pyspark/context.py
- python/pyspark/tests.py 2 additions, 2 deletionspython/pyspark/tests.py
- streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala 32 additions, 20 deletions...rc/main/scala/org/apache/spark/streaming/Checkpoint.scala
- streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala 18 additions, 14 deletions...n/scala/org/apache/spark/streaming/StreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala 8 additions, 5 deletions...pache/spark/streaming/api/java/JavaStreamingContext.scala
- streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala 92 additions, 61 deletions...org/apache/spark/streaming/dstream/FileInputDStream.scala
- streaming/src/main/scala/org/apache/spark/streaming/scheduler/JobGenerator.scala 48 additions, 19 deletions...a/org/apache/spark/streaming/scheduler/JobGenerator.scala
- streaming/src/test/scala/org/apache/spark/streaming/CheckpointSuite.scala 22 additions, 22 deletions...st/scala/org/apache/spark/streaming/CheckpointSuite.scala
- streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala 1 addition, 1 deletion.../scala/org/apache/spark/streaming/InputStreamsSuite.scala
Loading
Please register or sign in to comment