-
- Downloads
[SPARK-13738][SQL] Cleanup Data Source resolution
Follow-up to #11509, that simply refactors the interface that we use when resolving a pluggable `DataSource`. - Multiple functions share the same set of arguments so we make this a case class, called `DataSource`. Actual resolution is now done by calling a function on this class. - Instead of having multiple methods named `apply` (some of which do writing some of which do reading) we now explicitly have `resolveRelation()` and `write(mode, df)`. - Get rid of `Array[String]` since this is an internal API and was forcing us to awkwardly call `toArray` in a bunch of places. Author: Michael Armbrust <michael@databricks.com> Closes #11572 from marmbrus/dataSourceResolution.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala 18 additions, 16 deletions...src/main/scala/org/apache/spark/sql/DataFrameReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 15 additions, 14 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 73 additions, 95 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala 22 additions, 2 deletions...e/spark/sql/execution/datasources/PartitioningUtils.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala 11 additions, 10 deletions...cala/org/apache/spark/sql/execution/datasources/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala 5 additions, 9 deletions...la/org/apache/spark/sql/execution/datasources/rules.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala 1 addition, 1 deletion.../main/scala/org/apache/spark/sql/sources/interfaces.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala 7 additions, 7 deletions...ache/spark/sql/execution/datasources/json/JsonSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/ResolvedDataSourceSuite.scala 15 additions, 13 deletions...rg/apache/spark/sql/sources/ResolvedDataSourceSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 12 additions, 12 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala 16 additions, 15 deletions.../scala/org/apache/spark/sql/hive/execution/commands.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala 2 additions, 3 deletions...ain/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala
Loading
Please register or sign in to comment