Skip to content
Snippets Groups Projects
Commit 72edc7e9 authored by Burak Yavuz's avatar Burak Yavuz Committed by Yin Huai
Browse files

[SPARK-17531] Don't initialize Hive Listeners for the Execution Client

## What changes were proposed in this pull request?

If a user provides listeners inside the Hive Conf, the configuration for these listeners are passed to the Hive Execution Client as well. This may cause issues for two reasons:
1. The Execution Client will actually generate garbage
2. The listener class needs to be both in the Spark Classpath and Hive Classpath

This PR empties the listener configurations in `HiveUtils.newTemporaryConfiguration` so that the execution client will not contain the listener confs, but the metadata client will.

## How was this patch tested?

Unit tests

Author: Burak Yavuz <brkyvz@gmail.com>

Closes #15086 from brkyvz/null-listeners.
parent 4ba63b19
No related branches found
No related tags found
No related merge requests found
......@@ -394,6 +394,13 @@ private[spark] object HiveUtils extends Logging {
// hive.metastore.uris is not set.
propMap.put(ConfVars.METASTOREURIS.varname, "")
// The execution client will generate garbage events, therefore the listeners that are generated
// for the execution clients are useless. In order to not output garbage, we don't generate
// these listeners.
propMap.put(ConfVars.METASTORE_PRE_EVENT_LISTENERS.varname, "")
propMap.put(ConfVars.METASTORE_EVENT_LISTENERS.varname, "")
propMap.put(ConfVars.METASTORE_END_FUNCTION_LISTENERS.varname, "")
propMap.toMap
}
......
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.spark.sql.hive
import org.apache.hadoop.hive.conf.HiveConf.ConfVars
import org.apache.spark.sql.hive.test.TestHiveSingleton
import org.apache.spark.sql.test.SQLTestUtils
import org.apache.spark.sql.QueryTest
class HiveUtilsSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
test("newTemporaryConfiguration overwrites listener configurations") {
Seq(true, false).foreach { useInMemoryDerby =>
val conf = HiveUtils.newTemporaryConfiguration(useInMemoryDerby)
assert(conf(ConfVars.METASTORE_PRE_EVENT_LISTENERS.varname) === "")
assert(conf(ConfVars.METASTORE_EVENT_LISTENERS.varname) === "")
assert(conf(ConfVars.METASTORE_END_FUNCTION_LISTENERS.varname) === "")
}
}
}
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment