-
- Downloads
[SPARK-16031] Add debug-only socket source in Structured Streaming
## What changes were proposed in this pull request? This patch adds a text-based socket source similar to the one in Spark Streaming for debugging and tutorials. The source is clearly marked as debug-only so that users don't try to run it in production applications, because this type of source cannot provide HA without storing a lot of state in Spark. ## How was this patch tested? Unit tests and manual tests in spark-shell. Author: Matei Zaharia <matei@databricks.com> Closes #13748 from mateiz/socket-source.
Showing
- sql/core/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister 1 addition, 0 deletions.../services/org.apache.spark.sql.sources.DataSourceRegister
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala 2 additions, 0 deletions...ache/spark/sql/execution/streaming/FileStreamSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Source.scala 3 additions, 0 deletions...ala/org/apache/spark/sql/execution/streaming/Source.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 1 addition, 0 deletions...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/memory.scala 2 additions, 0 deletions...ala/org/apache/spark/sql/execution/streaming/memory.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/socket.scala 144 additions, 0 deletions...ala/org/apache/spark/sql/execution/streaming/socket.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/TextSocketStreamSuite.scala 136 additions, 0 deletions...spark/sql/execution/streaming/TextSocketStreamSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala 2 additions, 0 deletions...st/scala/org/apache/spark/sql/streaming/StreamSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala 2 additions, 0 deletions...park/sql/streaming/test/DataStreamReaderWriterSuite.scala
Loading
Please register or sign in to comment