Skip to content
Snippets Groups Projects
Commit 3e139e23 authored by 福星's avatar 福星 Committed by Reynold Xin
Browse files

[SPARK-18237][HIVE] hive.exec.stagingdir have no effect


hive.exec.stagingdir have no effect in spark2.0.1,
Hive confs in hive-site.xml will be loaded in `hadoopConf`, so we should use `hadoopConf` in `InsertIntoHiveTable` instead of `SessionState.conf`

Author: 福星 <fuxing@wacai.com>

Closes #15744 from ClassNotFoundExp/master.

(cherry picked from commit 16293311)
Signed-off-by: default avatarReynold Xin <rxin@databricks.com>
parent 4f91630c
No related branches found
No related tags found
No related merge requests found
......@@ -76,7 +76,8 @@ case class InsertIntoHiveTable(
def output: Seq[Attribute] = Seq.empty
val stagingDir = sessionState.conf.getConfString("hive.exec.stagingdir", ".hive-staging")
val hadoopConf = sessionState.newHadoopConf()
val stagingDir = hadoopConf.get("hive.exec.stagingdir", ".hive-staging")
private def executionId: String = {
val rand: Random = new Random
......@@ -163,7 +164,6 @@ case class InsertIntoHiveTable(
// instances within the closure, since Serializer is not serializable while TableDesc is.
val tableDesc = table.tableDesc
val tableLocation = table.hiveQlTable.getDataLocation
val hadoopConf = sessionState.newHadoopConf()
val tmpLocation = getExternalTmpPath(tableLocation, hadoopConf)
val fileSinkConf = new FileSinkDesc(tmpLocation.toString, tableDesc, false)
val isCompressed = hadoopConf.get("hive.exec.compress.output", "false").toBoolean
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment