-
- Downloads
[SPARK-15933][SQL][STREAMING] Refactored DF reader-writer to use readStream...
[SPARK-15933][SQL][STREAMING] Refactored DF reader-writer to use readStream and writeStream for streaming DFs ## What changes were proposed in this pull request? Currently, the DataFrameReader/Writer has method that are needed for streaming and non-streaming DFs. This is quite awkward because each method in them through runtime exception for one case or the other. So rather having half the methods throw runtime exceptions, its just better to have a different reader/writer API for streams. - [x] Python API!! ## How was this patch tested? Existing unit tests + two sets of unit tests for DataFrameReader/Writer and DataStreamReader/Writer. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #13653 from tdas/SPARK-15933.
Showing
- python/pyspark/sql/context.py 13 additions, 0 deletionspython/pyspark/sql/context.py
- python/pyspark/sql/dataframe.py 16 additions, 2 deletionspython/pyspark/sql/dataframe.py
- python/pyspark/sql/readwriter.py 498 additions, 142 deletionspython/pyspark/sql/readwriter.py
- python/pyspark/sql/session.py 16 additions, 1 deletionpython/pyspark/sql/session.py
- python/pyspark/sql/streaming.py 4 additions, 4 deletionspython/pyspark/sql/streaming.py
- python/pyspark/sql/tests.py 23 additions, 19 deletionspython/pyspark/sql/tests.py
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala 1 addition, 33 deletions...src/main/scala/org/apache/spark/sql/DataFrameReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 1 addition, 254 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 26 additions, 3 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 18 additions, 2 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 15 additions, 1 deletion...re/src/main/scala/org/apache/spark/sql/SparkSession.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala 294 additions, 0 deletions...ala/org/apache/spark/sql/streaming/DataStreamReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala 386 additions, 0 deletions...ala/org/apache/spark/sql/streaming/DataStreamWriter.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/ForeachSinkSuite.scala 4 additions, 3 deletions...ache/spark/sql/execution/streaming/ForeachSinkSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/ContinuousQueryManagerSuite.scala 2 additions, 2 deletions...che/spark/sql/streaming/ContinuousQueryManagerSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/ContinuousQuerySuite.scala 2 additions, 2 deletions...org/apache/spark/sql/streaming/ContinuousQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSinkSuite.scala 8 additions, 8 deletions.../org/apache/spark/sql/streaming/FileStreamSinkSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStreamSourceSuite.scala 6 additions, 6 deletions...rg/apache/spark/sql/streaming/FileStreamSourceSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/FileStressSuite.scala 5 additions, 5 deletions...cala/org/apache/spark/sql/streaming/FileStressSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/MemorySinkSuite.scala 12 additions, 12 deletions...cala/org/apache/spark/sql/streaming/MemorySinkSuite.scala
Loading
Please register or sign in to comment