-
- Downloads
[SPARK-20431][SS][FOLLOWUP] Specify a schema by using a DDL-formatted string in DataStreamReader
## What changes were proposed in this pull request? This pr supported a DDL-formatted string in `DataStreamReader.schema`. This fix could make users easily define a schema without importing the type classes. For example, ```scala scala> spark.readStream.schema("col0 INT, col1 DOUBLE").load("/tmp/abc").printSchema() root |-- col0: integer (nullable = true) |-- col1: double (nullable = true) ``` ## How was this patch tested? Added tests in `DataStreamReaderWriterSuite`. Author: hyukjinkwon <gurwls223@gmail.com> Closes #18373 from HyukjinKwon/SPARK-20431.
Showing
- python/pyspark/sql/readwriter.py 2 additions, 0 deletionspython/pyspark/sql/readwriter.py
- python/pyspark/sql/streaming.py 16 additions, 8 deletionspython/pyspark/sql/streaming.py
- sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala 12 additions, 0 deletions...ala/org/apache/spark/sql/streaming/DataStreamReader.scala
- sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala 12 additions, 0 deletions...park/sql/streaming/test/DataStreamReaderWriterSuite.scala
Loading
Please register or sign in to comment