-
- Downloads
[SPARK-20213][SQL] Fix DataFrameWriter operations in SQL UI tab
## What changes were proposed in this pull request? Currently the `DataFrameWriter` operations have several problems: 1. non-file-format data source writing action doesn't show up in the SQL tab in Spark UI 2. file-format data source writing action shows a scan node in the SQL tab, without saying anything about writing. (streaming also have this issue, but not fixed in this PR) 3. Spark SQL CLI actions don't show up in the SQL tab. This PR fixes all of them, by refactoring the `ExecuteCommandExec` to make it have children. close https://github.com/apache/spark/pull/17540 ## How was this patch tested? existing tests. Also test the UI manually. For a simple command: `Seq(1 -> "a").toDF("i", "j").write.parquet("/tmp/qwe")` before this PR: <img width="266" alt="qq20170523-035840 2x" src="https://cloud.githubusercontent.com/assets/3182036/26326050/24e18ba2-3f6c-11e7-8817-6dd275bf6ac5.png"> after this PR: <img width="287" alt="qq20170523-035708 2x" src="https://cloud.githubusercontent.com/assets/3182036/26326054/2ad7f460-3f6c-11e7-8053-d68325beb28f.png"> Author: Wenchen Fan <wenchen@databricks.com> Closes #18064 from cloud-fan/execution.
Showing
- external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaWriter.scala 4 additions, 6 deletions...ain/scala/org/apache/spark/sql/kafka010/KafkaWriter.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Command.scala 2 additions, 1 deletion...org/apache/spark/sql/catalyst/plans/logical/Command.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala 7 additions, 8 deletions.../apache/spark/sql/catalyst/plans/logical/Statistics.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 6 additions, 6 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 40 additions, 8 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala 4 additions, 3 deletions...scala/org/apache/spark/sql/execution/QueryExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala 13 additions, 0 deletions...n/scala/org/apache/spark/sql/execution/SQLExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala 1 addition, 1 deletion...cala/org/apache/spark/sql/execution/SparkStrategies.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala 1 addition, 1 deletion...pache/spark/sql/execution/columnar/InMemoryRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala 1 addition, 1 deletion.../spark/sql/execution/columnar/InMemoryTableScanExec.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeColumnCommand.scala 5 additions, 2 deletions...he/spark/sql/execution/command/AnalyzeColumnCommand.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/AnalyzeTableCommand.scala 1 addition, 1 deletion...che/spark/sql/execution/command/AnalyzeTableCommand.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/cache.scala 5 additions, 5 deletions.../scala/org/apache/spark/sql/execution/command/cache.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 17 additions, 7 deletions...ala/org/apache/spark/sql/execution/command/commands.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala 2 additions, 2 deletions.../spark/sql/execution/command/createDataSourceTables.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala 2 additions, 2 deletions.../scala/org/apache/spark/sql/execution/command/views.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 26 additions, 35 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala 49 additions, 49 deletions...he/spark/sql/execution/datasources/FileFormatWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoDataSourceCommand.scala 1 addition, 1 deletion...l/execution/datasources/InsertIntoDataSourceCommand.scala
Loading
Please register or sign in to comment