-
- Downloads
[SPARK-14555] First cut of Python API for Structured Streaming
## What changes were proposed in this pull request? This patch provides a first cut of python APIs for structured streaming. This PR provides the new classes: - ContinuousQuery - Trigger - ProcessingTime in pyspark under `pyspark.sql.streaming`. In addition, it contains the new methods added under: - `DataFrameWriter` a) `startStream` b) `trigger` c) `queryName` - `DataFrameReader` a) `stream` - `DataFrame` a) `isStreaming` This PR doesn't contain all methods exposed for `ContinuousQuery`, for example: - `exception` - `sourceStatuses` - `sinkStatus` They may be added in a follow up. This PR also contains some very minor doc fixes in the Scala side. ## How was this patch tested? Python doc tests TODO: - [ ] verify Python docs look good Author: Burak Yavuz <brkyvz@gmail.com> Author: Burak Yavuz <burak@databricks.com> Closes #12320 from brkyvz/stream-python.
Showing
- python/pyspark/__init__.py 15 additions, 0 deletionspython/pyspark/__init__.py
- python/pyspark/ml/classification.py 1 addition, 0 deletionspython/pyspark/ml/classification.py
- python/pyspark/ml/clustering.py 1 addition, 1 deletionpython/pyspark/ml/clustering.py
- python/pyspark/ml/evaluation.py 1 addition, 2 deletionspython/pyspark/ml/evaluation.py
- python/pyspark/ml/feature.py 2 additions, 2 deletionspython/pyspark/ml/feature.py
- python/pyspark/ml/pipeline.py 2 additions, 3 deletionspython/pyspark/ml/pipeline.py
- python/pyspark/ml/recommendation.py 1 addition, 1 deletionpython/pyspark/ml/recommendation.py
- python/pyspark/ml/regression.py 1 addition, 1 deletionpython/pyspark/ml/regression.py
- python/pyspark/ml/tests.py 1 addition, 1 deletionpython/pyspark/ml/tests.py
- python/pyspark/ml/tuning.py 2 additions, 2 deletionspython/pyspark/ml/tuning.py
- python/pyspark/ml/util.py 0 additions, 15 deletionspython/pyspark/ml/util.py
- python/pyspark/sql/dataframe.py 12 additions, 0 deletionspython/pyspark/sql/dataframe.py
- python/pyspark/sql/readwriter.py 120 additions, 1 deletionpython/pyspark/sql/readwriter.py
- python/pyspark/sql/streaming.py 124 additions, 0 deletionspython/pyspark/sql/streaming.py
- python/pyspark/sql/tests.py 93 additions, 0 deletionspython/pyspark/sql/tests.py
- python/test_support/sql/streaming/text-test.txt 2 additions, 0 deletionspython/test_support/sql/streaming/text-test.txt
- sql/core/src/main/scala/org/apache/spark/sql/ContinuousQuery.scala 2 additions, 2 deletions...src/main/scala/org/apache/spark/sql/ContinuousQuery.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 4 additions, 4 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/Trigger.scala 13 additions, 13 deletionssql/core/src/main/scala/org/apache/spark/sql/Trigger.scala
Loading
Please register or sign in to comment