-
- Downloads
[SPARK-16020][SQL] Fix complete mode aggregation with console sink
## What changes were proposed in this pull request? We cannot use `limit` on DataFrame in ConsoleSink because it will use a wrong planner. This PR just collects `DataFrame` and calls `show` on a batch DataFrame based on the result. This is fine since ConsoleSink is only for debugging. ## How was this patch tested? Manually confirmed ConsoleSink now works with complete mode aggregation. Author: Shixiong Zhu <shixiong@databricks.com> Closes #13740 from zsxwing/complete-console.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Sink.scala 3 additions, 0 deletions...scala/org/apache/spark/sql/execution/streaming/Sink.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/console.scala 3 additions, 1 deletion...la/org/apache/spark/sql/execution/streaming/console.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ConsoleSinkSuite.scala 99 additions, 0 deletions...ache/spark/sql/execution/streaming/ConsoleSinkSuite.scala
Loading
Please register or sign in to comment