Skip to content
Snippets Groups Projects
  • Joseph Batchik's avatar
    a3aec918
    [SPARK-9486][SQL] Add data source aliasing for external packages · a3aec918
    Joseph Batchik authored
    Users currently have to provide the full class name for external data sources, like:
    
    `sqlContext.read.format("com.databricks.spark.avro").load(path)`
    
    This allows external data source packages to register themselves using a Service Loader so that they can add custom alias like:
    
    `sqlContext.read.format("avro").load(path)`
    
    This makes it so that using external data source packages uses the same format as the internal data sources like parquet, json, etc.
    
    Author: Joseph Batchik <joseph.batchik@cloudera.com>
    Author: Joseph Batchik <josephbatchik@gmail.com>
    
    Closes #7802 from JDrit/service_loader and squashes the following commits:
    
    49a01ec [Joseph Batchik] fixed a couple of format / error bugs
    e5e93b2 [Joseph Batchik] modified rat file to only excluded added services
    72b349a [Joseph Batchik] fixed error with orc data source actually
    9f93ea7 [Joseph Batchik] fixed error with orc data source
    87b7f1c [Joseph Batchik] fixed typo
    101cd22 [Joseph Batchik] removing unneeded changes
    8f3cf43 [Joseph Batchik] merged in changes
    b63d337 [Joseph Batchik] merged in master
    95ae030 [Joseph Batchik] changed the new trait to be used as a mixin for data source to register themselves
    74db85e [Joseph Batchik] reformatted class loader
    ac2270d [Joseph Batchik] removing some added test
    a6926db [Joseph Batchik] added test cases for data source loader
    208a2a8 [Joseph Batchik] changes to do error catching if there are multiple data sources
    946186e [Joseph Batchik] started working on service loader
    a3aec918
    History
    [SPARK-9486][SQL] Add data source aliasing for external packages
    Joseph Batchik authored
    Users currently have to provide the full class name for external data sources, like:
    
    `sqlContext.read.format("com.databricks.spark.avro").load(path)`
    
    This allows external data source packages to register themselves using a Service Loader so that they can add custom alias like:
    
    `sqlContext.read.format("avro").load(path)`
    
    This makes it so that using external data source packages uses the same format as the internal data sources like parquet, json, etc.
    
    Author: Joseph Batchik <joseph.batchik@cloudera.com>
    Author: Joseph Batchik <josephbatchik@gmail.com>
    
    Closes #7802 from JDrit/service_loader and squashes the following commits:
    
    49a01ec [Joseph Batchik] fixed a couple of format / error bugs
    e5e93b2 [Joseph Batchik] modified rat file to only excluded added services
    72b349a [Joseph Batchik] fixed error with orc data source actually
    9f93ea7 [Joseph Batchik] fixed error with orc data source
    87b7f1c [Joseph Batchik] fixed typo
    101cd22 [Joseph Batchik] removing unneeded changes
    8f3cf43 [Joseph Batchik] merged in changes
    b63d337 [Joseph Batchik] merged in master
    95ae030 [Joseph Batchik] changed the new trait to be used as a mixin for data source to register themselves
    74db85e [Joseph Batchik] reformatted class loader
    ac2270d [Joseph Batchik] removing some added test
    a6926db [Joseph Batchik] added test cases for data source loader
    208a2a8 [Joseph Batchik] changes to do error catching if there are multiple data sources
    946186e [Joseph Batchik] started working on service loader