-
- Downloads
[SPARK-18497][SS] Make ForeachSink support watermark
## What changes were proposed in this pull request? The issue in ForeachSink is the new created DataSet still uses the old QueryExecution. When `foreachPartition` is called, `QueryExecution.toString` will be called and then fail because it doesn't know how to plan EventTimeWatermark. This PR just replaces the QueryExecution with IncrementalExecution to fix the issue. ## How was this patch tested? `test("foreach with watermark")`. Author: Shixiong Zhu <shixiong@databricks.com> Closes #15934 from zsxwing/SPARK-18497. (cherry picked from commit 2a40de40) Signed-off-by:Tathagata Das <tathagata.das1565@gmail.com>
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ForeachSink.scala 8 additions, 8 deletions...rg/apache/spark/sql/execution/streaming/ForeachSink.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ForeachSinkSuite.scala 35 additions, 0 deletions...ache/spark/sql/execution/streaming/ForeachSinkSuite.scala
Loading
Please register or sign in to comment