Skip to content
Snippets Groups Projects
Commit 3783539d authored by Kunal Khamar's avatar Kunal Khamar Committed by Tathagata Das
Browse files

[SPARK-19873][SS] Record num shuffle partitions in offset log and enforce in next batch.

## What changes were proposed in this pull request?

If the user changes the shuffle partition number between batches, Streaming aggregation will fail.

Here are some possible cases:

- Change "spark.sql.shuffle.partitions"
- Use "repartition" and change the partition number in codes
- RangePartitioner doesn't generate deterministic partitions. Right now it's safe as we disallow sort before aggregation. Not sure if we will add some operators using RangePartitioner in future.

## How was this patch tested?

- Unit tests
- Manual tests
  - forward compatibility tested by using the new `OffsetSeqMetadata` json with Spark v2.1.0

Author: Kunal Khamar <kkhamar@outlook.com>

Closes #17216 from kunalkhamar/num-partitions.
parent 7de66bae
No related branches found
No related tags found
No related merge requests found
Showing
with 63 additions and 20 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment