Skip to content
Snippets Groups Projects
Commit 6f61e1f9 authored by Cheng Lian's avatar Cheng Lian Committed by Patrick Wendell
Browse files

[SPARK-4761][SQL] Enables Kryo by default in Spark SQL Thrift server

Enables Kryo and disables reference tracking by default in Spark SQL Thrift server. Configurations explicitly defined by users in `spark-defaults.conf` are respected (the Thrift server is started by `spark-submit`, which handles configuration properties properly).

<!-- Reviewable:start -->
[<img src="https://reviewable.io/review_button.png" height=40 alt="Review on Reviewable"/>](https://reviewable.io/reviews/apache/spark/3621)
<!-- Reviewable:end -->

Author: Cheng Lian <lian@databricks.com>

Closes #3621 from liancheng/kryo-by-default and squashes the following commits:

70c2775 [Cheng Lian] Enables Kryo by default in Spark SQL Thrift server
parent f5801e81
No related branches found
No related tags found
No related merge requests found
...@@ -32,11 +32,21 @@ private[hive] object SparkSQLEnv extends Logging { ...@@ -32,11 +32,21 @@ private[hive] object SparkSQLEnv extends Logging {
def init() { def init() {
if (hiveContext == null) { if (hiveContext == null) {
val sparkConf = new SparkConf() val sparkConf = new SparkConf(loadDefaults = true)
val maybeSerializer = sparkConf.getOption("spark.serializer")
val maybeKryoReferenceTracking = sparkConf.getOption("spark.kryo.referenceTracking")
sparkConf
.setAppName(s"SparkSQL::${java.net.InetAddress.getLocalHost.getHostName}") .setAppName(s"SparkSQL::${java.net.InetAddress.getLocalHost.getHostName}")
.set("spark.sql.hive.version", HiveShim.version) .set("spark.sql.hive.version", HiveShim.version)
sparkContext = new SparkContext(sparkConf) .set(
"spark.serializer",
maybeSerializer.getOrElse("org.apache.spark.serializer.KryoSerializer"))
.set(
"spark.kryo.referenceTracking",
maybeKryoReferenceTracking.getOrElse("false"))
sparkContext = new SparkContext(sparkConf)
sparkContext.addSparkListener(new StatsReportListener()) sparkContext.addSparkListener(new StatsReportListener())
hiveContext = new HiveContext(sparkContext) hiveContext = new HiveContext(sparkContext)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment