-
- Downloads
[SPARK-14669] [SQL] Fix some SQL metrics in codegen and added more
## What changes were proposed in this pull request? 1. Fix the "spill size" of TungstenAggregate and Sort 2. Rename "data size" to "peak memory" to match the actual meaning (also consistent with task metrics) 3. Added "data size" for ShuffleExchange and BroadcastExchange 4. Added some timing for Sort, Aggregate and BroadcastExchange (this requires another patch to work) ## How was this patch tested? Existing tests.  Author: Davies Liu <davies@databricks.com> Closes #12425 from davies/fix_metrics.
Showing
- core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java 11 additions, 1 deletion...ark/util/collection/unsafe/sort/UnsafeExternalSorter.java
- sql/core/src/main/java/org/apache/spark/sql/execution/UnsafeKVExternalSorter.java 7 additions, 0 deletions...rg/apache/spark/sql/execution/UnsafeKVExternalSorter.java
- sql/core/src/main/scala/org/apache/spark/sql/execution/Sort.scala 11 additions, 8 deletions.../src/main/scala/org/apache/spark/sql/execution/Sort.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/UnsafeRowSerializer.scala 15 additions, 4 deletions.../org/apache/spark/sql/execution/UnsafeRowSerializer.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregate.scala 24 additions, 10 deletions...che/spark/sql/execution/aggregate/TungstenAggregate.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala 4 additions, 4 deletions...sql/execution/aggregate/TungstenAggregationIterator.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchange.scala 19 additions, 1 deletion...ache/spark/sql/execution/exchange/BroadcastExchange.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchange.scala 7 additions, 2 deletions...apache/spark/sql/execution/exchange/ShuffleExchange.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoin.scala 8 additions, 1 deletion...g/apache/spark/sql/execution/joins/ShuffledHashJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 4 additions, 1 deletion...la/org/apache/spark/sql/execution/metric/SQLMetrics.scala
Loading
Please register or sign in to comment