-
- Downloads
[SPARK-10810] [SPARK-10902] [SQL] Improve session management in SQL
This PR improve the sessions management by replacing the thread-local based to one SQLContext per session approach, introduce separated temporary tables and UDFs/UDAFs for each session. A new session of SQLContext could be created by: 1) create an new SQLContext 2) call newSession() on existing SQLContext For HiveContext, in order to reduce the cost for each session, the classloader and Hive client are shared across multiple sessions (created by newSession). CacheManager is also shared by multiple sessions, so cache a table multiple times in different sessions will not cause multiple copies of in-memory cache. Added jars are still shared by all the sessions, because SparkContext does not support sessions. cc marmbrus yhuai rxin Author: Davies Liu <davies@databricks.com> Closes #8909 from davies/sessions.
Showing
- project/MimaExcludes.scala 21 additions, 1 deletionproject/MimaExcludes.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala 21 additions, 7 deletions...apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 93 additions, 71 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala 3 additions, 11 deletions...n/scala/org/apache/spark/sql/execution/CacheManager.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLContextSuite.scala 39 additions, 20 deletions...src/test/scala/org/apache/spark/sql/SQLContextSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala 7 additions, 14 deletions...test/scala/org/apache/spark/sql/test/TestSQLContext.scala
- sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala 11 additions, 65 deletions...ql/hive/thriftserver/SparkExecuteStatementOperation.scala
- sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLSessionManager.scala 5 additions, 4 deletions.../spark/sql/hive/thriftserver/SparkSQLSessionManager.scala
- sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/server/SparkSQLOperationManager.scala 3 additions, 2 deletions...l/hive/thriftserver/server/SparkSQLOperationManager.scala
- sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala 4 additions, 4 deletions...ala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala
- sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala 40 additions, 36 deletions...spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 98 additions, 57 deletions...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 15 additions, 13 deletions...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientInterface.scala 9 additions, 0 deletions...la/org/apache/spark/sql/hive/client/ClientInterface.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala 48 additions, 37 deletions...cala/org/apache/spark/sql/hive/client/ClientWrapper.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala 65 additions, 42 deletions...g/apache/spark/sql/hive/client/IsolatedClientLoader.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala 4 additions, 23 deletions.../scala/org/apache/spark/sql/hive/execution/commands.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala 9 additions, 18 deletions.../main/scala/org/apache/spark/sql/hive/test/TestHive.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveQlSuite.scala 3 additions, 10 deletions...rc/test/scala/org/apache/spark/sql/hive/HiveQlSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala 3 additions, 3 deletions...cala/org/apache/spark/sql/hive/client/VersionsSuite.scala
Loading
Please register or sign in to comment