-
- Downloads
[SPARK-18908][SS] Creating StreamingQueryException should check if logicalPlan is created
## What changes were proposed in this pull request? This PR audits places using `logicalPlan` in StreamExecution and ensures they all handles the case that `logicalPlan` cannot be created. In addition, this PR also fixes the following issues in `StreamingQueryException`: - `StreamingQueryException` and `StreamExecution` are cycle-dependent because in the `StreamingQueryException`'s constructor, it calls `StreamExecution`'s `toDebugString` which uses `StreamingQueryException`. Hence it will output `null` value in the error message. - Duplicated stack trace when calling Throwable.printStackTrace because StreamingQueryException's toString contains the stack trace. ## How was this patch tested? The updated `test("max files per trigger - incorrect values")`. I found this issue when I switched from `testStream` to the real codes to verify the failure in this test. Author: Shixiong Zhu <shixiong@databricks.com> Closes #16322 from zsxwing/SPARK-18907.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 93 additions, 48 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryException.scala 5 additions, 23 deletions.../apache/spark/sql/streaming/StreamingQueryException.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala 25 additions, 14 deletions...rg/apache/spark/sql/streaming/FileStreamSourceSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala 2 additions, 1 deletion...st/scala/org/apache/spark/sql/streaming/StreamSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamTest.scala 37 additions, 15 deletions...est/scala/org/apache/spark/sql/streaming/StreamTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryListenerSuite.scala 1 addition, 1 deletion...che/spark/sql/streaming/StreamingQueryListenerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQuerySuite.scala 2 additions, 2 deletions.../org/apache/spark/sql/streaming/StreamingQuerySuite.scala
Loading
Please register or sign in to comment