-
- Downloads
[SPARK-6923] [SPARK-7550] [SQL] Persists data source relations in Hive...
[SPARK-6923] [SPARK-7550] [SQL] Persists data source relations in Hive compatible format when possible This PR is a fork of PR #5733 authored by chenghao-intel. For committers who's going to merge this PR, please set the author to "Cheng Hao <hao.chengintel.com>". ---- When a data source relation meets the following requirements, we persist it in Hive compatible format, so that other systems like Hive can access it: 1. It's a `HadoopFsRelation` 2. It has only one input path 3. It's non-partitioned 4. It's data source provider can be naturally mapped to a Hive builtin SerDe (e.g. ORC and Parquet) Author: Cheng Lian <lian@databricks.com> Author: Cheng Hao <hao.cheng@intel.com> Closes #7967 from liancheng/spark-6923/refactoring-pr-5733 and squashes the following commits: 5175ee6 [Cheng Lian] Fixes an oudated comment 3870166 [Cheng Lian] Fixes build error and comments 864acee [Cheng Lian] Refactors PR #5733 3490cdc [Cheng Hao] update the scaladoc 6f57669 [Cheng Hao] write schema info to hivemetastore for data source
Showing
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 37 additions, 16 deletionssql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 7 additions, 0 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 139 additions, 7 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 15 additions, 34 deletions...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala 4 additions, 2 deletions...ain/scala/org/apache/spark/sql/hive/orc/OrcRelation.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala 122 additions, 11 deletions...org/apache/spark/sql/hive/HiveMetastoreCatalogSuite.scala
Loading
Please register or sign in to comment