-
- Downloads
[SPARK-19540][SQL] Add ability to clone SparkSession wherein cloned session...
[SPARK-19540][SQL] Add ability to clone SparkSession wherein cloned session has an identical copy of the SessionState Forking a newSession() from SparkSession currently makes a new SparkSession that does not retain SessionState (i.e. temporary tables, SQL config, registered functions etc.) This change adds a method cloneSession() which creates a new SparkSession with a copy of the parent's SessionState. Subsequent changes to base session are not propagated to cloned session, clone is independent after creation. If the base is changed after clone has been created, say user registers new UDF, then the new UDF will not be available inside the clone. Same goes for configs and temp tables. Unit tests Author: Kunal Khamar <kkhamar@outlook.com> Author: Shixiong Zhu <shixiong@databricks.com> Closes #16826 from kunalkhamar/fork-sparksession.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystConf.scala 6 additions, 1 deletion...in/scala/org/apache/spark/sql/catalyst/CatalystConf.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 4 additions, 1 deletion...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala 35 additions, 3 deletions...rg/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala 55 additions, 0 deletions...ache/spark/sql/catalyst/catalog/SessionCatalogSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/ExperimentalMethods.scala 6 additions, 0 deletions...main/scala/org/apache/spark/sql/ExperimentalMethods.scala
- sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 45 additions, 14 deletions...re/src/main/scala/org/apache/spark/sql/SparkSession.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala 2 additions, 1 deletion...la/org/apache/spark/sql/execution/datasources/rules.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 8 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SessionState.scala 151 additions, 84 deletions...in/scala/org/apache/spark/sql/internal/SessionState.scala
- sql/core/src/test/scala/org/apache/spark/sql/SessionStateSuite.scala 162 additions, 0 deletions...c/test/scala/org/apache/spark/sql/SessionStateSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala 20 additions, 1 deletion...st/scala/org/apache/spark/sql/internal/CatalogSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfEntrySuite.scala 18 additions, 0 deletions...ala/org/apache/spark/sql/internal/SQLConfEntrySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala 9 additions, 11 deletions...test/scala/org/apache/spark/sql/test/TestSQLContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 3 additions, 2 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala 75 additions, 17 deletions.../scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionState.scala 191 additions, 70 deletions...in/scala/org/apache/spark/sql/hive/HiveSessionState.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala 2 additions, 0 deletions...ala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala 36 additions, 31 deletions.../main/scala/org/apache/spark/sql/hive/test/TestHive.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSessionCatalogSuite.scala 112 additions, 0 deletions...a/org/apache/spark/sql/hive/HiveSessionCatalogSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSessionStateSuite.scala 41 additions, 0 deletions...ala/org/apache/spark/sql/hive/HiveSessionStateSuite.scala
Loading
Please register or sign in to comment