Skip to content
Snippets Groups Projects
  • Forest Fang's avatar
    94e2064f
    [SPARK-11244][SPARKR] sparkR.stop() should remove SQLContext · 94e2064f
    Forest Fang authored
    SparkR should remove `.sparkRSQLsc` and `.sparkRHivesc` when `sparkR.stop()` is called. Otherwise even when SparkContext is reinitialized, `sparkRSQL.init` returns the stale copy of the object and complains:
    
    ```r
    sc <- sparkR.init("local")
    sqlContext <- sparkRSQL.init(sc)
    sparkR.stop()
    sc <- sparkR.init("local")
    sqlContext <- sparkRSQL.init(sc)
    sqlContext
    ```
    producing
    ```r
    Error in callJMethod(x, "getClass") :
      Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed.
    ```
    
    I have added the check and removal only when SparkContext itself is initialized. I have also added corresponding test for this fix. Let me know if you want me to move the test to SQL test suite instead.
    
    p.s. I tried lint-r but ended up a lots of errors on existing code.
    
    Author: Forest Fang <forest.fang@outlook.com>
    
    Closes #9205 from saurfang/sparkR.stop.
    94e2064f
    History
    [SPARK-11244][SPARKR] sparkR.stop() should remove SQLContext
    Forest Fang authored
    SparkR should remove `.sparkRSQLsc` and `.sparkRHivesc` when `sparkR.stop()` is called. Otherwise even when SparkContext is reinitialized, `sparkRSQL.init` returns the stale copy of the object and complains:
    
    ```r
    sc <- sparkR.init("local")
    sqlContext <- sparkRSQL.init(sc)
    sparkR.stop()
    sc <- sparkR.init("local")
    sqlContext <- sparkRSQL.init(sc)
    sqlContext
    ```
    producing
    ```r
    Error in callJMethod(x, "getClass") :
      Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed.
    ```
    
    I have added the check and removal only when SparkContext itself is initialized. I have also added corresponding test for this fix. Let me know if you want me to move the test to SQL test suite instead.
    
    p.s. I tried lint-r but ended up a lots of errors on existing code.
    
    Author: Forest Fang <forest.fang@outlook.com>
    
    Closes #9205 from saurfang/sparkR.stop.
test_context.R 1.94 KiB