Skip to content
Snippets Groups Projects
Commit 10b3ca3e authored by Yuming Wang's avatar Yuming Wang Committed by gatorsmile
Browse files

[SPARK-21574][SQL] Point out user to set hive config before SparkSession is initialized

## What changes were proposed in this pull request?
Since Spark 2.0.0, SET hive config commands do not pass the values to HiveClient, this PR point out user to set hive config before SparkSession is initialized when they try to set hive config.

## How was this patch tested?
manual tests

<img width="1637" alt="spark-set" src="https://user-images.githubusercontent.com/5399861/29001141-03f943ee-7ab3-11e7-8584-ba5a5e81f6ad.png">

Author: Yuming Wang <wgyumg@gmail.com>

Closes #18769 from wangyum/SPARK-21574.
parent d4e7f20f
No related branches found
No related tags found
No related merge requests found
......@@ -21,6 +21,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.catalyst.expressions.Attribute
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.StaticSQLConf.CATALOG_IMPLEMENTATION
import org.apache.spark.sql.types.{StringType, StructField, StructType}
......@@ -87,6 +88,14 @@ case class SetCommand(kv: Option[(String, Option[String])]) extends RunnableComm
// Configures a single property.
case Some((key, Some(value))) =>
val runFunc = (sparkSession: SparkSession) => {
if (sparkSession.conf.get(CATALOG_IMPLEMENTATION.key).equals("hive") &&
key.startsWith("hive.")) {
logWarning(s"'SET $key=$value' might not work, since Spark doesn't support changing " +
"the Hive config dynamically. Please passing the Hive-specific config by adding the " +
s"prefix spark.hadoop (e.g., spark.hadoop.$key) when starting a Spark application. " +
"For details, see the link: https://spark.apache.org/docs/latest/configuration.html#" +
"dynamically-loading-spark-properties.")
}
sparkSession.conf.set(key, value)
Seq(Row(key, value))
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment