-
- Downloads
[SPARK-9486][SQL] Add data source aliasing for external packages
Users currently have to provide the full class name for external data sources, like: `sqlContext.read.format("com.databricks.spark.avro").load(path)` This allows external data source packages to register themselves using a Service Loader so that they can add custom alias like: `sqlContext.read.format("avro").load(path)` This makes it so that using external data source packages uses the same format as the internal data sources like parquet, json, etc. Author: Joseph Batchik <joseph.batchik@cloudera.com> Author: Joseph Batchik <josephbatchik@gmail.com> Closes #7802 from JDrit/service_loader and squashes the following commits: 49a01ec [Joseph Batchik] fixed a couple of format / error bugs e5e93b2 [Joseph Batchik] modified rat file to only excluded added services 72b349a [Joseph Batchik] fixed error with orc data source actually 9f93ea7 [Joseph Batchik] fixed error with orc data source 87b7f1c [Joseph Batchik] fixed typo 101cd22 [Joseph Batchik] removing unneeded changes 8f3cf43 [Joseph Batchik] merged in changes b63d337 [Joseph Batchik] merged in master 95ae030 [Joseph Batchik] changed the new trait to be used as a mixin for data source to register themselves 74db85e [Joseph Batchik] reformatted class loader ac2270d [Joseph Batchik] removing some added test a6926db [Joseph Batchik] added test cases for data source loader 208a2a8 [Joseph Batchik] changes to do error catching if there are multiple data sources 946186e [Joseph Batchik] started working on service loader
Showing
- .rat-excludes 1 addition, 0 deletions.rat-excludes
- sql/core/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister 3 additions, 0 deletions.../services/org.apache.spark.sql.sources.DataSourceRegister
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/ddl.scala 26 additions, 26 deletions...cala/org/apache/spark/sql/execution/datasources/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/jdbc/JDBCRelation.scala 4 additions, 1 deletion...c/main/scala/org/apache/spark/sql/jdbc/JDBCRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala 4 additions, 1 deletion...c/main/scala/org/apache/spark/sql/json/JSONRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetRelation.scala 4 additions, 1 deletion.../scala/org/apache/spark/sql/parquet/ParquetRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/interfaces.scala 21 additions, 0 deletions.../main/scala/org/apache/spark/sql/sources/interfaces.scala
- sql/core/src/test/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister 3 additions, 0 deletions.../services/org.apache.spark.sql.sources.DataSourceRegister
- sql/core/src/test/scala/org/apache/spark/sql/sources/DDLSourceLoadSuite.scala 85 additions, 0 deletions...ala/org/apache/spark/sql/sources/DDLSourceLoadSuite.scala
- sql/hive/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister 1 addition, 0 deletions.../services/org.apache.spark.sql.sources.DataSourceRegister
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala 4 additions, 1 deletion...ain/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala
Loading
Please register or sign in to comment