-
- Downloads
[SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to...
[SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to the existing Scala SparkContext's SparkConf ## What changes were proposed in this pull request? When we create a SparkSession at the Python side, it is possible that a SparkContext has been created. For this case, we need to set configs of the SparkSession builder to the Scala SparkContext's SparkConf (we need to do so because conf changes on a active Python SparkContext will not be propagated to the JVM side). Otherwise, we may create a wrong SparkSession (e.g. Hive support is not enabled even if enableHiveSupport is called). ## How was this patch tested? New tests and manual tests. Author: Yin Huai <yhuai@databricks.com> Closes #13931 from yhuai/SPARK-16224.
Showing
- python/pyspark/context.py 2 additions, 0 deletionspython/pyspark/context.py
- python/pyspark/sql/session.py 7 additions, 0 deletionspython/pyspark/sql/session.py
- python/pyspark/sql/tests.py 42 additions, 1 deletionpython/pyspark/sql/tests.py
- python/pyspark/tests.py 8 additions, 0 deletionspython/pyspark/tests.py
Loading
Please register or sign in to comment