Skip to content
Snippets Groups Projects
Commit d1b80ab9 authored by Wenchen Fan's avatar Wenchen Fan
Browse files

[SPARK-20967][SQL] SharedState.externalCatalog is not really lazy

## What changes were proposed in this pull request?

`SharedState.externalCatalog` is marked as a `lazy val` but actually it's not lazy. We access `externalCatalog` while initializing `SharedState` and thus eliminate the effort of `lazy val`. When creating `ExternalCatalog` we will try to connect to the metastore and may throw an error, so it makes sense to make it a `lazy val` in `SharedState`.

## How was this patch tested?

existing tests.

Author: Wenchen Fan <wenchen@databricks.com>

Closes #18187 from cloud-fan/minor.
parent 625cebfd
No related branches found
No related tags found
No related merge requests found
...@@ -90,38 +90,38 @@ private[sql] class SharedState(val sparkContext: SparkContext) extends Logging { ...@@ -90,38 +90,38 @@ private[sql] class SharedState(val sparkContext: SparkContext) extends Logging {
/** /**
* A catalog that interacts with external systems. * A catalog that interacts with external systems.
*/ */
lazy val externalCatalog: ExternalCatalog = lazy val externalCatalog: ExternalCatalog = {
SharedState.reflect[ExternalCatalog, SparkConf, Configuration]( val externalCatalog = SharedState.reflect[ExternalCatalog, SparkConf, Configuration](
SharedState.externalCatalogClassName(sparkContext.conf), SharedState.externalCatalogClassName(sparkContext.conf),
sparkContext.conf, sparkContext.conf,
sparkContext.hadoopConfiguration) sparkContext.hadoopConfiguration)
// Create the default database if it doesn't exist.
{
val defaultDbDefinition = CatalogDatabase( val defaultDbDefinition = CatalogDatabase(
SessionCatalog.DEFAULT_DATABASE, SessionCatalog.DEFAULT_DATABASE,
"default database", "default database",
CatalogUtils.stringToURI(warehousePath), CatalogUtils.stringToURI(warehousePath),
Map()) Map())
// Initialize default database if it doesn't exist // Create default database if it doesn't exist
if (!externalCatalog.databaseExists(SessionCatalog.DEFAULT_DATABASE)) { if (!externalCatalog.databaseExists(SessionCatalog.DEFAULT_DATABASE)) {
// There may be another Spark application creating default database at the same time, here we // There may be another Spark application creating default database at the same time, here we
// set `ignoreIfExists = true` to avoid `DatabaseAlreadyExists` exception. // set `ignoreIfExists = true` to avoid `DatabaseAlreadyExists` exception.
externalCatalog.createDatabase(defaultDbDefinition, ignoreIfExists = true) externalCatalog.createDatabase(defaultDbDefinition, ignoreIfExists = true)
} }
}
// Make sure we propagate external catalog events to the spark listener bus // Make sure we propagate external catalog events to the spark listener bus
externalCatalog.addListener(new ExternalCatalogEventListener { externalCatalog.addListener(new ExternalCatalogEventListener {
override def onEvent(event: ExternalCatalogEvent): Unit = { override def onEvent(event: ExternalCatalogEvent): Unit = {
sparkContext.listenerBus.post(event) sparkContext.listenerBus.post(event)
} }
}) })
externalCatalog
}
/** /**
* A manager for global temporary views. * A manager for global temporary views.
*/ */
val globalTempViewManager: GlobalTempViewManager = { lazy val globalTempViewManager: GlobalTempViewManager = {
// System preserved database should not exists in metastore. However it's hard to guarantee it // System preserved database should not exists in metastore. However it's hard to guarantee it
// for every session, because case-sensitivity differs. Here we always lowercase it to make our // for every session, because case-sensitivity differs. Here we always lowercase it to make our
// life easier. // life easier.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment