-
- Downloads
[SPARK-18927][SS] MemorySink for StructuredStreaming can't recover from...
[SPARK-18927][SS] MemorySink for StructuredStreaming can't recover from checkpoint if location is provided in SessionConf ## What changes were proposed in this pull request? Checkpoint Location can be defined for a StructuredStreaming on a per-query basis by the `DataStreamWriter` options, but it can also be provided through SparkSession configurations. It should be able to recover in both cases when the OutputMode is Complete for MemorySinks. ## How was this patch tested? Unit tests Author: Burak Yavuz <brkyvz@gmail.com> Closes #16342 from brkyvz/chk-rec.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/streaming/DataStreamWriter.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala 24 additions, 8 deletions...park/sql/streaming/test/DataStreamReaderWriterSuite.scala
Loading
Please register or sign in to comment