-
- Downloads
[SPARK-19873][SS] Record num shuffle partitions in offset log and enforce in next batch.
## What changes were proposed in this pull request? If the user changes the shuffle partition number between batches, Streaming aggregation will fail. Here are some possible cases: - Change "spark.sql.shuffle.partitions" - Use "repartition" and change the partition number in codes - RangePartitioner doesn't generate deterministic partitions. Right now it's safe as we disallow sort before aggregation. Not sure if we will add some operators using RangePartitioner in future. ## How was this patch tested? - Unit tests - Manual tests - forward compatibility tested by using the new `OffsetSeqMetadata` json with Spark v2.1.0 Author: Kunal Khamar <kkhamar@outlook.com> Closes #17216 from kunalkhamar/num-partitions.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala 5 additions, 3 deletions.../org/apache/spark/sql/execution/streaming/OffsetSeq.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 47 additions, 13 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala 4 additions, 4 deletions...rg/apache/spark/sql/streaming/StreamingQueryManager.scala
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/metadata 1 addition, 0 deletions...es/structured-streaming/checkpoint-version-2.1.0/metadata
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/offsets/0 3 additions, 0 deletions...s/structured-streaming/checkpoint-version-2.1.0/offsets/0
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/offsets/1 3 additions, 0 deletions...s/structured-streaming/checkpoint-version-2.1.0/offsets/1
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/0/1.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/0/1.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/0/2.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/0/2.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/1/1.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/1/1.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/1/2.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/1/2.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/2/1.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/2/1.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/2/2.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/2/2.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/3/1.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/3/1.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/3/2.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/3/2.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/4/1.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/4/1.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/4/2.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/4/2.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/5/1.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/5/1.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/5/2.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/5/2.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/6/1.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/6/1.delta
- sql/core/src/test/resources/structured-streaming/checkpoint-version-2.1.0/state/0/6/2.delta 0 additions, 0 deletions...ured-streaming/checkpoint-version-2.1.0/state/0/6/2.delta
Loading
Please register or sign in to comment