-
- Downloads
[SPARK-13055] SQLHistoryListener throws ClassCastException
This is an existing issue uncovered recently by #10835. The reason for the exception was because the `SQLHistoryListener` gets all sorts of accumulators, not just the ones that represent SQL metrics. For example, the listener gets the `internal.metrics.shuffleRead.remoteBlocksFetched`, which is an Int, then it proceeds to cast the Int to a Long, which fails. The fix is to mark accumulators representing SQL metrics using some internal metadata. Then we can identify which ones are SQL metrics and only process those in the `SQLHistoryListener`. Author: Andrew Or <andrew@databricks.com> Closes #10971 from andrewor14/fix-sql-history.
Showing
- core/src/main/scala/org/apache/spark/Accumulable.scala 8 additions, 0 deletionscore/src/main/scala/org/apache/spark/Accumulable.scala
- core/src/main/scala/org/apache/spark/executor/TaskMetrics.scala 2 additions, 2 deletions...rc/main/scala/org/apache/spark/executor/TaskMetrics.scala
- core/src/main/scala/org/apache/spark/scheduler/AccumulableInfo.scala 4 additions, 1 deletion...in/scala/org/apache/spark/scheduler/AccumulableInfo.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 2 additions, 5 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 4 additions, 2 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- core/src/test/scala/org/apache/spark/executor/TaskMetricsSuite.scala 1 addition, 3 deletions...st/scala/org/apache/spark/executor/TaskMetricsSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala 4 additions, 10 deletions.../scala/org/apache/spark/scheduler/DAGSchedulerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 2 additions, 6 deletions...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
- core/src/test/scala/org/apache/spark/util/JsonProtocolSuite.scala 10 additions, 6 deletions.../test/scala/org/apache/spark/util/JsonProtocolSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 20 additions, 1 deletion...la/org/apache/spark/sql/execution/metric/SQLMetrics.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala 16 additions, 7 deletions...scala/org/apache/spark/sql/execution/ui/SQLListener.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala 23 additions, 1 deletion...g/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLListenerSuite.scala 37 additions, 1 deletion.../org/apache/spark/sql/execution/ui/SQLListenerSuite.scala
Loading
Please register or sign in to comment