-
- Downloads
[SPARK-11206] Support SQL UI on the history server
On the live web UI, there is a SQL tab which provides valuable information for the SQL query. But once the workload is finished, we won't see the SQL tab on the history server. It will be helpful if we support SQL UI on the history server so we can analyze it even after its execution. To support SQL UI on the history server: 1. I added an `onOtherEvent` method to the `SparkListener` trait and post all SQL related events to the same event bus. 2. Two SQL events `SparkListenerSQLExecutionStart` and `SparkListenerSQLExecutionEnd` are defined in the sql module. 3. The new SQL events are written to event log using Jackson. 4. A new trait `SparkHistoryListenerFactory` is added to allow the history server to feed events to the SQL history listener. The SQL implementation is loaded at runtime using `java.util.ServiceLoader`. Author: Carson Wang <carson.wang@intel.com> Closes #9297 from carsonwang/SqlHistoryUI.
Showing
- .rat-excludes 1 addition, 0 deletions.rat-excludes
- core/src/main/java/org/apache/spark/JavaSparkListener.java 3 additions, 0 deletionscore/src/main/java/org/apache/spark/JavaSparkListener.java
- core/src/main/java/org/apache/spark/SparkFirehoseListener.java 4 additions, 0 deletions...src/main/java/org/apache/spark/SparkFirehoseListener.java
- core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala 4 additions, 0 deletions...ala/org/apache/spark/scheduler/EventLoggingListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala 22 additions, 2 deletions...main/scala/org/apache/spark/scheduler/SparkListener.scala
- core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala 1 addition, 0 deletions...n/scala/org/apache/spark/scheduler/SparkListenerBus.scala
- core/src/main/scala/org/apache/spark/ui/SparkUI.scala 14 additions, 2 deletionscore/src/main/scala/org/apache/spark/ui/SparkUI.scala
- core/src/main/scala/org/apache/spark/util/JsonProtocol.scala 9 additions, 2 deletionscore/src/main/scala/org/apache/spark/util/JsonProtocol.scala
- sql/core/src/main/resources/META-INF/services/org.apache.spark.scheduler.SparkHistoryListenerFactory 1 addition, 0 deletions...es/org.apache.spark.scheduler.SparkHistoryListenerFactory
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 14 additions, 4 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala 7 additions, 17 deletions...n/scala/org/apache/spark/sql/execution/SQLExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala 46 additions, 0 deletions.../scala/org/apache/spark/sql/execution/SparkPlanInfo.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetricInfo.scala 30 additions, 0 deletions...org/apache/spark/sql/execution/metric/SQLMetricInfo.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLMetrics.scala 34 additions, 22 deletions...la/org/apache/spark/sql/execution/metric/SQLMetrics.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala 1 addition, 3 deletions...ala/org/apache/spark/sql/execution/ui/ExecutionPage.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala 97 additions, 42 deletions...scala/org/apache/spark/sql/execution/ui/SQLListener.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLTab.scala 1 addition, 11 deletions...main/scala/org/apache/spark/sql/execution/ui/SQLTab.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala 10 additions, 10 deletions...la/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsSuite.scala 3 additions, 1 deletion...g/apache/spark/sql/execution/metric/SQLMetricsSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLListenerSuite.scala 24 additions, 19 deletions.../org/apache/spark/sql/execution/ui/SQLListenerSuite.scala
Loading
Please register or sign in to comment