-
- Downloads
[SPARK-12012][SQL] Show more comprehensive PhysicalRDD metadata when visualizing SQL query plan
This PR adds a `private[sql]` method `metadata` to `SparkPlan`, which can be used to describe detail information about a physical plan during visualization. Specifically, this PR uses this method to provide details of `PhysicalRDD`s translated from a data source relation. For example, a `ParquetRelation` converted from Hive metastore table `default.psrc` is now shown as the following screenshot:  And here is the screenshot for a regular `ParquetRelation` (not converted from Hive metastore table) loaded from a really long path:  Author: Cheng Lian <lian@databricks.com> Closes #10004 from liancheng/spark-12012.physical-rdd-metadata.
Showing
- python/pyspark/sql/dataframe.py 1 addition, 1 deletionpython/pyspark/sql/dataframe.py
- sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala 14 additions, 5 deletions...in/scala/org/apache/spark/sql/execution/ExistingRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 5 additions, 0 deletions...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala 2 additions, 1 deletion.../scala/org/apache/spark/sql/execution/SparkPlanInfo.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 1 addition, 1 deletion...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala 19 additions, 3 deletions.../spark/sql/execution/datasources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRelation.scala 10 additions, 0 deletions...k/sql/execution/datasources/parquet/ParquetRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala 27 additions, 18 deletions...la/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala 1 addition, 1 deletion.../main/scala/org/apache/spark/sql/sources/interfaces.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/PlannerSuite.scala 1 addition, 1 deletion...t/scala/org/apache/spark/sql/execution/PlannerSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 6 additions, 1 deletion...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
Loading
Please register or sign in to comment