-
- Downloads
[SPARK-15887][SQL] Bring back the hive-site.xml support for Spark 2.0
## What changes were proposed in this pull request? Right now, Spark 2.0 does not load hive-site.xml. Based on users' feedback, it seems make sense to still load this conf file. This PR adds a `hadoopConf` API in `SharedState`, which is `sparkContext.hadoopConfiguration` by default. When users are under hive context, `SharedState.hadoopConf` will load hive-site.xml and append its configs to `sparkContext.hadoopConfiguration`. When we need to read hadoop config in spark sql, we should call `SessionState.newHadoopConf`, which contains `sparkContext.hadoopConfiguration`, hive-site.xml and sql configs. ## How was this patch tested? new test in `HiveDataFrameSuite` Author: Wenchen Fan <wenchen@databricks.com> Closes #13611 from cloud-fan/hive-site.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala 1 addition, 1 deletion...sql/execution/datasources/parquet/ParquetFileFormat.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala 1 addition, 1 deletion...in/scala/org/apache/spark/sql/internal/SessionState.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala 17 additions, 2 deletions...ain/scala/org/apache/spark/sql/internal/SharedState.scala
- sql/core/src/test/resources/hive-site.xml 26 additions, 0 deletionssql/core/src/test/resources/hive-site.xml
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 4 additions, 0 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSharedState.scala 2 additions, 3 deletions...ain/scala/org/apache/spark/sql/hive/HiveSharedState.scala
- sql/hive/src/test/resources/hive-site.xml 26 additions, 0 deletionssql/hive/src/test/resources/hive-site.xml
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala 5 additions, 0 deletions.../scala/org/apache/spark/sql/hive/HiveDataFrameSuite.scala
Loading
Please register or sign in to comment