-
- Downloads
[SPARK-14473][SQL] Define analysis rules to catch operations not supported in streaming
## What changes were proposed in this pull request? There are many operations that are currently not supported in the streaming execution. For example: - joining two streams - unioning a stream and a batch source - sorting - window functions (not time windows) - distinct aggregates Furthermore, executing a query with a stream source as a batch query should also fail. This patch add an additional step after analysis in the QueryExecution which will check that all the operations in the analyzed logical plan is supported or not. ## How was this patch tested? unit tests. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #12246 from tdas/SPARK-14473.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/OutputMode.scala 23 additions, 0 deletions...a/org/apache/spark/sql/catalyst/analysis/OutputMode.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala 145 additions, 0 deletions...k/sql/catalyst/analysis/UnsupportedOperationChecker.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala 18 additions, 0 deletions...ain/scala/org/apache/spark/sql/catalyst/dsl/package.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala 3 additions, 0 deletions...apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationsSuite.scala 384 additions, 0 deletions...rk/sql/catalyst/analysis/UnsupportedOperationsSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/LogicalPlanSuite.scala 21 additions, 0 deletions...rg/apache/spark/sql/catalyst/plans/LogicalPlanSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/ContinuousQueryManager.scala 14 additions, 2 deletions...n/scala/org/apache/spark/sql/ContinuousQueryManager.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 1 addition, 3 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala 9 additions, 0 deletions...scala/org/apache/spark/sql/execution/QueryExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala 7 additions, 1 deletion.../spark/sql/execution/streaming/IncrementalExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 9 additions, 2 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamingRelation.scala 2 additions, 0 deletions...che/spark/sql/execution/streaming/StreamingRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 8 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/test/scala/org/apache/spark/sql/StreamTest.scala 7 additions, 1 deletion...core/src/test/scala/org/apache/spark/sql/StreamTest.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala 31 additions, 3 deletions...st/scala/org/apache/spark/sql/streaming/StreamSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingAggregationSuite.scala 3 additions, 0 deletions...pache/spark/sql/streaming/StreamingAggregationSuite.scala
Loading
Please register or sign in to comment