-
- Downloads
[SPARK-13080][SQL] Implement new Catalog API using Hive
## What changes were proposed in this pull request? This is a step towards merging `SQLContext` and `HiveContext`. A new internal Catalog API was introduced in #10982 and extended in #11069. This patch introduces an implementation of this API using `HiveClient`, an existing interface to Hive. It also extends `HiveClient` with additional calls to Hive that are needed to complete the catalog implementation. *Where should I start reviewing?* The new catalog introduced is `HiveCatalog`. This class is relatively simple because it just calls `HiveClientImpl`, where most of the new logic is. I would not start with `HiveClient`, `HiveQl`, or `HiveMetastoreCatalog`, which are modified mainly because of a refactor. *Why is this patch so big?* I had to refactor HiveClient to remove an intermediate representation of databases, tables, partitions etc. After this refactor `CatalogTable` convert directly to and from `HiveTable` (etc.). Otherwise we would have to first convert `CatalogTable` to the intermediate representation and then convert that to HiveTable, which is messy. The new class hierarchy is as follows: ``` org.apache.spark.sql.catalyst.catalog.Catalog - org.apache.spark.sql.catalyst.catalog.InMemoryCatalog - org.apache.spark.sql.hive.HiveCatalog ``` Note that, as of this patch, none of these classes are currently used anywhere yet. This will come in the future before the Spark 2.0 release. ## How was the this patch tested? All existing unit tests, and HiveCatalogSuite that extends CatalogTestCases. Author: Andrew Or <andrew@databricks.com> Author: Reynold Xin <rxin@databricks.com> Closes #11293 from rxin/hive-catalog.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/AnalysisException.scala 3 additions, 0 deletions...c/main/scala/org/apache/spark/sql/AnalysisException.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Catalog.scala 0 additions, 9 deletions...cala/org/apache/spark/sql/catalyst/analysis/Catalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala 52 additions, 0 deletions...che/spark/sql/catalyst/analysis/NoSuchItemException.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala 80 additions, 74 deletions...g/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala 124 additions, 66 deletions...ala/org/apache/spark/sql/catalyst/catalog/interface.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/CatalogTestCases.scala 193 additions, 91 deletions.../apache/spark/sql/catalyst/catalog/CatalogTestCases.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveCatalog.scala 293 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/hive/HiveCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 94 additions, 77 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 75 additions, 70 deletions...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClient.scala 134 additions, 76 deletions...n/scala/org/apache/spark/sql/hive/client/HiveClient.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 259 additions, 120 deletions...ala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateTableAsSelect.scala 10 additions, 10 deletions...apache/spark/sql/hive/execution/CreateTableAsSelect.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateViewAsSelect.scala 12 additions, 8 deletions.../apache/spark/sql/hive/execution/CreateViewAsSelect.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala 1 addition, 1 deletion...apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveCatalogSuite.scala 49 additions, 0 deletions...st/scala/org/apache/spark/sql/hive/HiveCatalogSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala 20 additions, 20 deletions...org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveQlSuite.scala 48 additions, 46 deletions...rc/test/scala/org/apache/spark/sql/hive/HiveQlSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala 15 additions, 10 deletions...org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/MultiDatabaseSuite.scala 2 additions, 2 deletions.../scala/org/apache/spark/sql/hive/MultiDatabaseSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala 18 additions, 19 deletions...cala/org/apache/spark/sql/hive/client/VersionsSuite.scala
Loading
Please register or sign in to comment