-
- Downloads
[SPARK-20100][SQL] Refactor SessionState initialization
## What changes were proposed in this pull request? The current SessionState initialization code path is quite complex. A part of the creation is done in the SessionState companion objects, a part of the creation is one inside the SessionState class, and a part is done by passing functions. This PR refactors this code path, and consolidates SessionState initialization into a builder class. This SessionState will not do any initialization and just becomes a place holder for the various Spark SQL internals. This also lays the ground work for two future improvements: 1. This provides us with a start for removing the `HiveSessionState`. Removing the `HiveSessionState` would also require us to move resource loading into a separate class, and to (re)move metadata hive. 2. This makes it easier to customize the Spark Session. Currently you will need to create a custom version of the builder. I have added hooks to facilitate this. A future step will be to create a semi stable API on top of this. ## How was this patch tested? Existing tests. Author: Herman van Hovell <hvanhovell@databricks.com> Closes #17433 from hvanhovell/SPARK-20100.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala 15 additions, 31 deletions...rg/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 8 additions, 8 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala 8 additions, 14 deletions...ache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala 10 additions, 2 deletions...scala/org/apache/spark/sql/execution/SparkOptimizer.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanner.scala 9 additions, 2 deletions...n/scala/org/apache/spark/sql/execution/SparkPlanner.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/IncrementalExecution.scala 10 additions, 13 deletions.../spark/sql/execution/streaming/IncrementalExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala 44 additions, 136 deletions...in/scala/org/apache/spark/sql/internal/SessionState.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/sessionStateBuilders.scala 279 additions, 0 deletions.../org/apache/spark/sql/internal/sessionStateBuilders.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala 12 additions, 11 deletions...test/scala/org/apache/spark/sql/test/TestSQLContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala 6 additions, 70 deletions.../scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala 114 additions, 145 deletions...in/scala/org/apache/spark/sql/hive/HiveSessionState.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala 32 additions, 28 deletions.../main/scala/org/apache/spark/sql/hive/test/TestHive.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSessionCatalogSuite.scala 0 additions, 112 deletions...a/org/apache/spark/sql/hive/HiveSessionCatalogSuite.scala
Loading
Please register or sign in to comment