Skip to content
Snippets Groups Projects
Commit 1904bac3 authored by Cheng Hao's avatar Cheng Hao Committed by Michael Armbrust
Browse files

[SPARK-3392] [SQL] Show value spark.sql.shuffle.partitions for mapred.reduce.tasks

This is a tiny fix for getting the value of "mapred.reduce.tasks", which make more sense for the hive user.
As well as the command "set -v", which should output verbose information for all of the key/values.

Author: Cheng Hao <hao.cheng@intel.com>

Closes #2261 from chenghao-intel/set_mapreduce_tasks and squashes the following commits:

653858a [Cheng Hao] show value spark.sql.shuffle.partitions for mapred.reduce.tasks
parent ee575f12
No related branches found
No related tags found
No related merge requests found
...@@ -60,10 +60,10 @@ case class SetCommand( ...@@ -60,10 +60,10 @@ case class SetCommand(
logWarning(s"Property ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS} is deprecated, " + logWarning(s"Property ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS} is deprecated, " +
s"automatically converted to ${SQLConf.SHUFFLE_PARTITIONS} instead.") s"automatically converted to ${SQLConf.SHUFFLE_PARTITIONS} instead.")
context.setConf(SQLConf.SHUFFLE_PARTITIONS, v) context.setConf(SQLConf.SHUFFLE_PARTITIONS, v)
Array(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=$v")) Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=$v"))
} else { } else {
context.setConf(k, v) context.setConf(k, v)
Array(Row(s"$k=$v")) Seq(Row(s"$k=$v"))
} }
// Query the value bound to key k. // Query the value bound to key k.
...@@ -78,11 +78,19 @@ case class SetCommand( ...@@ -78,11 +78,19 @@ case class SetCommand(
"hive-hwi-0.12.0.jar", "hive-hwi-0.12.0.jar",
"hive-0.12.0.jar").mkString(":") "hive-0.12.0.jar").mkString(":")
Array( context.getAllConfs.map { case (k, v) =>
Row(s"$k=$v")
}.toSeq ++ Seq(
Row("system:java.class.path=" + hiveJars), Row("system:java.class.path=" + hiveJars),
Row("system:sun.java.command=shark.SharkServer2")) Row("system:sun.java.command=shark.SharkServer2"))
} else { } else {
Array(Row(s"$k=${context.getConf(k, "<undefined>")}")) if (k == SQLConf.Deprecated.MAPRED_REDUCE_TASKS) {
logWarning(s"Property ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS} is deprecated, " +
s"showing ${SQLConf.SHUFFLE_PARTITIONS} instead.")
Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=${context.numShufflePartitions}"))
} else {
Seq(Row(s"$k=${context.getConf(k, "<undefined>")}"))
}
} }
// Query all key-value pairs that are set in the SQLConf of the context. // Query all key-value pairs that are set in the SQLConf of the context.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment