Skip to content
Snippets Groups Projects
Commit 2b36f494 authored by Huaxin Gao's avatar Huaxin Gao Committed by Wenchen Fan
Browse files

[SPARK-17460][SQL] Make sure sizeInBytes in Statistics will not overflow


## What changes were proposed in this pull request?

1. In SparkStrategies.canBroadcast, I will add the check   plan.statistics.sizeInBytes >= 0
2. In LocalRelations.statistics, when calculate the statistics, I will change the size to BigInt so it won't overflow.

## How was this patch tested?

I will add a test case to make sure the statistics.sizeInBytes won't overflow.

Author: Huaxin Gao <huaxing@us.ibm.com>

Closes #16175 from huaxingao/spark-17460.

(cherry picked from commit c5172568)
Signed-off-by: default avatarWenchen Fan <wenchen@databricks.com>
parent b020ce40
No related branches found
No related tags found
No related merge requests found
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment