Skip to content
Snippets Groups Projects
  • Yin Huai's avatar
    0923c4f5
    [SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to... · 0923c4f5
    Yin Huai authored
    [SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to the existing Scala SparkContext's SparkConf
    
    ## What changes were proposed in this pull request?
    When we create a SparkSession at the Python side, it is possible that a SparkContext has been created. For this case, we need to set configs of the SparkSession builder to the Scala SparkContext's SparkConf (we need to do so because conf changes on a active Python SparkContext will not be propagated to the JVM side). Otherwise, we may create a wrong SparkSession (e.g. Hive support is not enabled even if enableHiveSupport is called).
    
    ## How was this patch tested?
    New tests and manual tests.
    
    Author: Yin Huai <yhuai@databricks.com>
    
    Closes #13931 from yhuai/SPARK-16224.
    0923c4f5
    History
    [SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to...
    Yin Huai authored
    [SPARK-16224] [SQL] [PYSPARK] SparkSession builder's configs need to be set to the existing Scala SparkContext's SparkConf
    
    ## What changes were proposed in this pull request?
    When we create a SparkSession at the Python side, it is possible that a SparkContext has been created. For this case, we need to set configs of the SparkSession builder to the Scala SparkContext's SparkConf (we need to do so because conf changes on a active Python SparkContext will not be propagated to the JVM side). Otherwise, we may create a wrong SparkSession (e.g. Hive support is not enabled even if enableHiveSupport is called).
    
    ## How was this patch tested?
    New tests and manual tests.
    
    Author: Yin Huai <yhuai@databricks.com>
    
    Closes #13931 from yhuai/SPARK-16224.