-
- Downloads
[SPARK-21147][SS] Throws an analysis exception when a user-specified schema is...
[SPARK-21147][SS] Throws an analysis exception when a user-specified schema is given in socket/rate sources ## What changes were proposed in this pull request? This PR proposes to throw an exception if a schema is provided by user to socket source as below: **socket source** ```scala import org.apache.spark.sql.types._ val userSpecifiedSchema = StructType( StructField("name", StringType) :: StructField("area", StringType) :: Nil) val df = spark.readStream.format("socket").option("host", "localhost").option("port", 9999).schema(userSpecifiedSchema).load df.printSchema ``` Before ``` root |-- value: string (nullable = true) ``` After ``` org.apache.spark.sql.AnalysisException: The socket source does not support a user-specified schema.; at org.apache.spark.sql.execution.streaming.TextSocketSourceProvider.sourceSchema(socket.scala:199) at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:192) at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:87) at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:87) at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30) at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:150) ... 50 elided ``` **rate source** ```scala spark.readStream.format("rate").schema(spark.range(1).schema).load().printSchema() ``` Before ``` root |-- timestamp: timestamp (nullable = true) |-- value: long (nullable = true)` ``` After ``` org.apache.spark.sql.AnalysisException: The rate source does not support a user-specified schema.; at org.apache.spark.sql.execution.streaming.RateSourceProvider.sourceSchema(RateSourceProvider.scala:57) at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:192) at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:87) at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:87) at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:30) at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:150) ... 48 elided ``` ## How was this patch tested? Unit test in `TextSocketStreamSuite` and `RateSourceSuite`. Author: hyukjinkwon <gurwls223@gmail.com> Closes #18365 from HyukjinKwon/SPARK-21147.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/RateSourceProvider.scala 7 additions, 2 deletions...he/spark/sql/execution/streaming/RateSourceProvider.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/socket.scala 6 additions, 2 deletions...ala/org/apache/spark/sql/execution/streaming/socket.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/RateSourceSuite.scala 12 additions, 0 deletions...pache/spark/sql/execution/streaming/RateSourceSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/TextSocketStreamSuite.scala 15 additions, 0 deletions...spark/sql/execution/streaming/TextSocketStreamSuite.scala
Loading
Please register or sign in to comment