Skip to content
Snippets Groups Projects
Commit 25b97589 authored by Tathagata Das's avatar Tathagata Das
Browse files

[SPARK-18834][SS] Expose event time stats through StreamingQueryProgress


## What changes were proposed in this pull request?

- Changed `StreamingQueryProgress.watermark` to `StreamingQueryProgress.queryTimestamps` which is a `Map[String, String]` containing the following keys: "eventTime.max", "eventTime.min", "eventTime.avg", "processingTime", "watermark". All of them UTC formatted strings.

- Renamed `StreamingQuery.timestamp` to `StreamingQueryProgress.triggerTimestamp` to differentiate from `queryTimestamps`. It has the timestamp of when the trigger was started.

## How was this patch tested?

Updated tests

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #16258 from tdas/SPARK-18834.

(cherry picked from commit c68fb426)
Signed-off-by: default avatarTathagata Das <tathagata.das1565@gmail.com>
parent f672bfdf
No related branches found
No related tags found
No related merge requests found
Showing with 161 additions and 66 deletions
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment