-
- Downloads
[SPARK-15792][SQL] Allows operator to change the verbosity in explain output
## What changes were proposed in this pull request? This PR allows customization of verbosity in explain output. After change, `dataframe.explain()` and `dataframe.explain(true)` has different verbosity output for physical plan. Currently, this PR only enables verbosity string for operator `HashAggregateExec` and `SortAggregateExec`. We will gradually enable verbosity string for more operators in future. **Less verbose mode:** dataframe.explain(extended = false) `output=[count(a)#85L]` is **NOT** displayed for HashAggregate. ``` scala> Seq((1,2,3)).toDF("a", "b", "c").createTempView("df2") scala> spark.sql("select count(a) from df2").explain() == Physical Plan == *HashAggregate(key=[], functions=[count(1)]) +- Exchange SinglePartition +- *HashAggregate(key=[], functions=[partial_count(1)]) +- LocalTableScan ``` **Verbose mode:** dataframe.explain(extended = true) `output=[count(a)#85L]` is displayed for HashAggregate. ``` scala> spark.sql("select count(a) from df2").explain(true) // "output=[count(a)#85L]" is added ... == Physical Plan == *HashAggregate(key=[], functions=[count(1)], output=[count(a)#85L]) +- Exchange SinglePartition +- *HashAggregate(key=[], functions=[partial_count(1)], output=[count#87L]) +- LocalTableScan ``` ## How was this patch tested? Manual test. Author: Sean Zhong <seanzhong@databricks.com> Closes #13535 from clockfly/verbose_breakdown_2.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala 4 additions, 0 deletions...rg/apache/spark/sql/catalyst/expressions/Expression.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 2 additions, 0 deletions...scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala 17 additions, 6 deletions.../scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala 8 additions, 6 deletions...scala/org/apache/spark/sql/execution/QueryExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala 4 additions, 2 deletions...rg/apache/spark/sql/execution/WholeStageCodegenExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala 10 additions, 2 deletions...che/spark/sql/execution/aggregate/HashAggregateExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala 10 additions, 2 deletions...che/spark/sql/execution/aggregate/SortAggregateExec.scala
Loading
Please register or sign in to comment