-
- Downloads
[SPARK-18671][SS][TEST] Added tests to ensure stability of that all Structured...
[SPARK-18671][SS][TEST] Added tests to ensure stability of that all Structured Streaming log formats ## What changes were proposed in this pull request? To be able to restart StreamingQueries across Spark version, we have already made the logs (offset log, file source log, file sink log) use json. We should added tests with actual json files in the Spark such that any incompatible changes in reading the logs is immediately caught. This PR add tests for FileStreamSourceLog, FileStreamSinkLog, and OffsetSeqLog. ## How was this patch tested? new unit tests Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #16128 from tdas/SPARK-18671. (cherry picked from commit 1ef6b296) Signed-off-by:Shixiong Zhu <shixiong@databricks.com>
Showing
- dev/.rat-excludes 1 addition, 0 deletionsdev/.rat-excludes
- external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala 8 additions, 1 deletion.../main/scala/org/apache/spark/sql/kafka010/JsonUtils.scala
- external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaSourceOffsetSuite.scala 12 additions, 0 deletions...rg/apache/spark/sql/kafka010/KafkaSourceOffsetSuite.scala
- sql/core/src/test/resources/structured-streaming/file-sink-log-version-2.1.0/7.compact 9 additions, 0 deletions...tructured-streaming/file-sink-log-version-2.1.0/7.compact
- sql/core/src/test/resources/structured-streaming/file-sink-log-version-2.1.0/8 3 additions, 0 deletions...ources/structured-streaming/file-sink-log-version-2.1.0/8
- sql/core/src/test/resources/structured-streaming/file-sink-log-version-2.1.0/9 2 additions, 0 deletions...ources/structured-streaming/file-sink-log-version-2.1.0/9
- sql/core/src/test/resources/structured-streaming/file-source-log-version-2.1.0/2.compact 4 additions, 0 deletions...uctured-streaming/file-source-log-version-2.1.0/2.compact
- sql/core/src/test/resources/structured-streaming/file-source-log-version-2.1.0/3 2 additions, 0 deletions...rces/structured-streaming/file-source-log-version-2.1.0/3
- sql/core/src/test/resources/structured-streaming/file-source-log-version-2.1.0/4 2 additions, 0 deletions...rces/structured-streaming/file-source-log-version-2.1.0/4
- sql/core/src/test/resources/structured-streaming/file-source-offset-version-2.1.0.txt 1 addition, 0 deletions...structured-streaming/file-source-offset-version-2.1.0.txt
- sql/core/src/test/resources/structured-streaming/kafka-source-offset-version-2.1.0.txt 1 addition, 0 deletions...tructured-streaming/kafka-source-offset-version-2.1.0.txt
- sql/core/src/test/resources/structured-streaming/offset-log-version-2.1.0/0 4 additions, 0 deletions...resources/structured-streaming/offset-log-version-2.1.0/0
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/FileStreamSinkLogSuite.scala 21 additions, 0 deletions...park/sql/execution/streaming/FileStreamSinkLogSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala 16 additions, 0 deletions...che/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala 28 additions, 2 deletions...rg/apache/spark/sql/streaming/FileStreamSourceSuite.scala
Loading
Please register or sign in to comment