-
- Downloads
[SPARK-15075][SPARK-15345][SQL] Clean up SparkSession builder and propagate...
[SPARK-15075][SPARK-15345][SQL] Clean up SparkSession builder and propagate config options to existing sessions if specified ## What changes were proposed in this pull request? Currently SparkSession.Builder use SQLContext.getOrCreate. It should probably the the other way around, i.e. all the core logic goes in SparkSession, and SQLContext just calls that. This patch does that. This patch also makes sure config options specified in the builder are propagated to the existing (and of course the new) SparkSession. ## How was this patch tested? Updated tests to reflect the change, and also introduced a new SparkSessionBuilderSuite that should cover all the branches. Author: Reynold Xin <rxin@databricks.com> Closes #13200 from rxin/SPARK-15075.
Showing
- mllib/src/test/java/org/apache/spark/ml/util/JavaDefaultReadWriteSuite.java 1 addition, 1 deletion...a/org/apache/spark/ml/util/JavaDefaultReadWriteSuite.java
- python/pyspark/sql/context.py 4 additions, 1 deletionpython/pyspark/sql/context.py
- python/pyspark/sql/session.py 14 additions, 3 deletionspython/pyspark/sql/session.py
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 1 addition, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 12 additions, 112 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 126 additions, 29 deletions...re/src/main/scala/org/apache/spark/sql/SparkSession.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala 2 additions, 1 deletion...in/scala/org/apache/spark/sql/execution/ExistingRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala 2 additions, 2 deletions...scala/org/apache/spark/sql/execution/QueryExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala 3 additions, 3 deletions...main/scala/org/apache/spark/sql/execution/SparkPlan.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 7 additions, 6 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/fileSourceInterfaces.scala 1 addition, 1 deletion...park/sql/execution/datasources/fileSourceInterfaces.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala 1 addition, 1 deletion...e/spark/sql/execution/datasources/jdbc/JDBCRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala 1 addition, 1 deletion...pache/spark/sql/execution/streaming/StreamExecution.scala
- sql/core/src/main/scala/org/apache/spark/sql/functions.scala 1 addition, 1 deletionsql/core/src/main/scala/org/apache/spark/sql/functions.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 0 additions, 10 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala 16 additions, 3 deletions...ain/scala/org/apache/spark/sql/internal/SharedState.scala
- sql/core/src/test/scala/org/apache/spark/sql/ListTablesSuite.scala 7 additions, 6 deletions...src/test/scala/org/apache/spark/sql/ListTablesSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala 0 additions, 100 deletions...st/scala/org/apache/spark/sql/MultiSQLContextsSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLContextSuite.scala 1 addition, 1 deletion...src/test/scala/org/apache/spark/sql/SQLContextSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 4 additions, 4 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
Loading
Please register or sign in to comment