-
- Downloads
[SPARK-19937] Collect metrics for remote bytes read to disk during shuffle.
In current code(https://github.com/apache/spark/pull/16989), big blocks are shuffled to disk. This pr proposes to collect metrics for remote bytes fetched to disk. Author: jinxing <jinxing6042@126.com> Closes #18249 from jinxing64/SPARK-19937.
Showing
- core/src/main/scala/org/apache/spark/InternalAccumulator.scala 1 addition, 0 deletions...src/main/scala/org/apache/spark/InternalAccumulator.scala
- core/src/main/scala/org/apache/spark/executor/ShuffleReadMetrics.scala 13 additions, 0 deletions.../scala/org/apache/spark/executor/ShuffleReadMetrics.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 1 addition, 0 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/status/api/v1/AllStagesResource.scala 2 additions, 0 deletions...la/org/apache/spark/status/api/v1/AllStagesResource.scala
- core/src/main/scala/org/apache/spark/status/api/v1/api.scala 2 additions, 0 deletionscore/src/main/scala/org/apache/spark/status/api/v1/api.scala
- core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala 6 additions, 0 deletions...rg/apache/spark/storage/ShuffleBlockFetcherIterator.scala
- core/src/main/scala/org/apache/spark/ui/jobs/UIData.scala 3 additions, 1 deletioncore/src/main/scala/org/apache/spark/ui/jobs/UIData.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 3 additions, 0 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/resources/HistoryServerExpectations/one_stage_attempt_json_expectation.json 8 additions, 0 deletions...erverExpectations/one_stage_attempt_json_expectation.json
- core/src/test/resources/HistoryServerExpectations/one_stage_json_expectation.json 8 additions, 0 deletions...HistoryServerExpectations/one_stage_json_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_expectation.json 20 additions, 0 deletions...istoryServerExpectations/stage_task_list_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_from_multi_attempt_app_json_1__expectation.json 8 additions, 0 deletions...task_list_from_multi_attempt_app_json_1__expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_from_multi_attempt_app_json_2__expectation.json 8 additions, 0 deletions...task_list_from_multi_attempt_app_json_2__expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_w__offset___length_expectation.json 50 additions, 0 deletions...tions/stage_task_list_w__offset___length_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_w__sortBy_expectation.json 20 additions, 0 deletions...erExpectations/stage_task_list_w__sortBy_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_w__sortBy_short_names___runtime_expectation.json 20 additions, 0 deletions...ask_list_w__sortBy_short_names___runtime_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_list_w__sortBy_short_names__runtime_expectation.json 20 additions, 0 deletions...task_list_w__sortBy_short_names__runtime_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_summary_w__custom_quantiles_expectation.json 1 addition, 0 deletions...s/stage_task_summary_w__custom_quantiles_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_summary_w_shuffle_read_expectation.json 1 addition, 0 deletions...ations/stage_task_summary_w_shuffle_read_expectation.json
- core/src/test/resources/HistoryServerExpectations/stage_task_summary_w_shuffle_write_expectation.json 1 addition, 0 deletions...tions/stage_task_summary_w_shuffle_write_expectation.json
Loading
Please register or sign in to comment