-
- Downloads
[SPARK-17492][SQL] Fix Reading Cataloged Data Sources without Extending SchemaRelationProvider
### What changes were proposed in this pull request? For data sources without extending `SchemaRelationProvider`, we expect users to not specify schemas when they creating tables. If the schema is input from users, an exception is issued. Since Spark 2.1, for any data source, to avoid infer the schema every time, we store the schema in the metastore catalog. Thus, when reading a cataloged data source table, the schema could be read from metastore catalog. In this case, we also got an exception. For example, ```Scala sql( s""" |CREATE TABLE relationProvierWithSchema |USING org.apache.spark.sql.sources.SimpleScanSource |OPTIONS ( | From '1', | To '10' |) """.stripMargin) spark.table(tableName).show() ``` ``` org.apache.spark.sql.sources.SimpleScanSource does not allow user-specified schemas.; ``` This PR is to fix the above issue. When building a data source, we introduce a flag `isSchemaFromUsers` to indicate whether the schema is really input from users. If true, we issue an exception. Otherwise, we will call the `createRelation` of `RelationProvider` to generate the `BaseRelation`, in which it contains the actual schema. ### How was this patch tested? Added a few cases. Author: gatorsmile <gatorsmile@gmail.com> Closes #15046 from gatorsmile/tempViewCases.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 7 additions, 2 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala 20 additions, 0 deletions...test/scala/org/apache/spark/sql/sources/InsertSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/TableScanSuite.scala 42 additions, 22 deletions...t/scala/org/apache/spark/sql/sources/TableScanSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala 33 additions, 0 deletions...rg/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
Loading
Please register or sign in to comment