Skip to content
Snippets Groups Projects
Commit e3cec18e authored by Ryan Blue's avatar Ryan Blue Committed by Marcelo Vanzin
Browse files

[SPARK-20084][CORE] Remove internal.metrics.updatedBlockStatuses from history files.


## What changes were proposed in this pull request?

Remove accumulator updates for internal.metrics.updatedBlockStatuses from SparkListenerTaskEnd entries in the history file. These can cause history files to grow to hundreds of GB because the value of the accumulator contains all tracked blocks.

## How was this patch tested?

Current History UI tests cover use of the history file.

Author: Ryan Blue <blue@apache.org>

Closes #17412 from rdblue/SPARK-20084-remove-block-accumulator-info.

(cherry picked from commit c4c03eed)
Signed-off-by: default avatarMarcelo Vanzin <vanzin@cloudera.com>
parent 6a1b2eb4
No related branches found
No related tags found
No related merge requests found
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment