Skip to content
Snippets Groups Projects
  • Jeff Zhang's avatar
    ea662286
    [SPARK-17261] [PYSPARK] Using HiveContext after re-creating SparkContext in... · ea662286
    Jeff Zhang authored
    [SPARK-17261] [PYSPARK] Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext"
    
    ## What changes were proposed in this pull request?
    
    Set SparkSession._instantiatedContext as None so that we can recreate SparkSession again.
    
    ## How was this patch tested?
    
    Tested manually using the following command in pyspark shell
    ```
    spark.stop()
    spark = SparkSession.builder.enableHiveSupport().getOrCreate()
    spark.sql("show databases").show()
    ```
    
    Author: Jeff Zhang <zjffdu@apache.org>
    
    Closes #14857 from zjffdu/SPARK-17261.
    ea662286
    History
    [SPARK-17261] [PYSPARK] Using HiveContext after re-creating SparkContext in...
    Jeff Zhang authored
    [SPARK-17261] [PYSPARK] Using HiveContext after re-creating SparkContext in Spark 2.0 throws "Java.lang.illegalStateException: Cannot call methods on a stopped sparkContext"
    
    ## What changes were proposed in this pull request?
    
    Set SparkSession._instantiatedContext as None so that we can recreate SparkSession again.
    
    ## How was this patch tested?
    
    Tested manually using the following command in pyspark shell
    ```
    spark.stop()
    spark = SparkSession.builder.enableHiveSupport().getOrCreate()
    spark.sql("show databases").show()
    ```
    
    Author: Jeff Zhang <zjffdu@apache.org>
    
    Closes #14857 from zjffdu/SPARK-17261.
session.py 24.25 KiB