-
- Downloads
[SPARK-14474][SQL] Move FileSource offset log into checkpointLocation
## What changes were proposed in this pull request?
Now that we have a single location for storing checkpointed state. This PR just propagates the checkpoint location into FileStreamSource so that we don't have one random log off on its own.
## How was this patch tested?
test("metadataPath should be in checkpointLocation")
Author: Shixiong Zhu <shixiong@databricks.com>
Closes #12247 from zsxwing/file-source-log-location.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/ContinuousQueryManager.scala 4 additions, 1 deletion...n/scala/org/apache/spark/sql/ContinuousQueryManager.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 42 additions, 20 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingRelation.scala 2 additions, 2 deletions...che/spark/sql/execution/streaming/StreamingRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala 9 additions, 0 deletions.../main/scala/org/apache/spark/sql/sources/interfaces.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/DataFrameReaderWriterSuite.scala 69 additions, 4 deletions...ache/spark/sql/streaming/DataFrameReaderWriterSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala 5 additions, 5 deletions...rg/apache/spark/sql/streaming/FileStreamSourceSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/MemorySinkSuite.scala 1 addition, 1 deletion...cala/org/apache/spark/sql/streaming/MemorySinkSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala 9 additions, 0 deletions...st/scala/org/apache/spark/sql/streaming/StreamSuite.scala
Loading
Please register or sign in to comment