Skip to content
Snippets Groups Projects
Commit c8c4441d authored by Ryan Blue's avatar Ryan Blue Committed by Wenchen Fan
Browse files

[SPARK-23418][SQL] Fail DataSourceV2 reads when user schema is passed, but not supported.

## What changes were proposed in this pull request?

DataSourceV2 initially allowed user-supplied schemas when a source doesn't implement `ReadSupportWithSchema`, as long as the schema was identical to the source's schema. This is confusing behavior because changes to an underlying table can cause a previously working job to fail with an exception that user-supplied schemas are not allowed.

This reverts commit adcb25a0624, which was added to #20387 so that it could be removed in a separate JIRA issue and PR.

## How was this patch tested?

Existing tests.

Author: Ryan Blue <blue@apache.org>

Closes #20603 from rdblue/SPARK-23418-revert-adcb25a0624.
parent 95e25ed1
No related branches found
No related tags found
No related merge requests found
...@@ -174,13 +174,6 @@ object DataSourceV2Relation { ...@@ -174,13 +174,6 @@ object DataSourceV2Relation {
v2Options: DataSourceOptions, v2Options: DataSourceOptions,
userSchema: Option[StructType]): StructType = { userSchema: Option[StructType]): StructType = {
val reader = userSchema match { val reader = userSchema match {
// TODO: remove this case because it is confusing for users
case Some(s) if !source.isInstanceOf[ReadSupportWithSchema] =>
val reader = source.asReadSupport.createReader(v2Options)
if (reader.readSchema() != s) {
throw new AnalysisException(s"${source.name} does not allow user-specified schemas.")
}
reader
case Some(s) => case Some(s) =>
source.asReadSupportWithSchema.createReader(s, v2Options) source.asReadSupportWithSchema.createReader(s, v2Options)
case _ => case _ =>
...@@ -195,11 +188,7 @@ object DataSourceV2Relation { ...@@ -195,11 +188,7 @@ object DataSourceV2Relation {
filters: Option[Seq[Expression]] = None, filters: Option[Seq[Expression]] = None,
userSpecifiedSchema: Option[StructType] = None): DataSourceV2Relation = { userSpecifiedSchema: Option[StructType] = None): DataSourceV2Relation = {
val projection = schema(source, makeV2Options(options), userSpecifiedSchema).toAttributes val projection = schema(source, makeV2Options(options), userSpecifiedSchema).toAttributes
DataSourceV2Relation(source, options, projection, filters, DataSourceV2Relation(source, options, projection, filters, userSpecifiedSchema)
// if the source does not implement ReadSupportWithSchema, then the userSpecifiedSchema must
// be equal to the reader's schema. the schema method enforces this. because the user schema
// and the reader's schema are identical, drop the user schema.
if (source.isInstanceOf[ReadSupportWithSchema]) userSpecifiedSchema else None)
} }
private def pushRequiredColumns(reader: DataSourceReader, struct: StructType): Unit = { private def pushRequiredColumns(reader: DataSourceReader, struct: StructType): Unit = {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment