Skip to content
Snippets Groups Projects
Commit 94e2064f authored by Forest Fang's avatar Forest Fang Committed by Shivaram Venkataraman
Browse files

[SPARK-11244][SPARKR] sparkR.stop() should remove SQLContext

SparkR should remove `.sparkRSQLsc` and `.sparkRHivesc` when `sparkR.stop()` is called. Otherwise even when SparkContext is reinitialized, `sparkRSQL.init` returns the stale copy of the object and complains:

```r
sc <- sparkR.init("local")
sqlContext <- sparkRSQL.init(sc)
sparkR.stop()
sc <- sparkR.init("local")
sqlContext <- sparkRSQL.init(sc)
sqlContext
```
producing
```r
Error in callJMethod(x, "getClass") :
  Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed.
```

I have added the check and removal only when SparkContext itself is initialized. I have also added corresponding test for this fix. Let me know if you want me to move the test to SQL test suite instead.

p.s. I tried lint-r but ended up a lots of errors on existing code.

Author: Forest Fang <forest.fang@outlook.com>

Closes #9205 from saurfang/sparkR.stop.
parent c03b6d11
No related branches found
No related tags found
No related merge requests found
......@@ -39,6 +39,14 @@ sparkR.stop <- function() {
sc <- get(".sparkRjsc", envir = env)
callJMethod(sc, "stop")
rm(".sparkRjsc", envir = env)
if (exists(".sparkRSQLsc", envir = env)) {
rm(".sparkRSQLsc", envir = env)
}
if (exists(".sparkRHivesc", envir = env)) {
rm(".sparkRHivesc", envir = env)
}
}
if (exists(".backendLaunched", envir = env)) {
......
......@@ -26,6 +26,16 @@ test_that("repeatedly starting and stopping SparkR", {
}
})
test_that("repeatedly starting and stopping SparkR SQL", {
for (i in 1:4) {
sc <- sparkR.init()
sqlContext <- sparkRSQL.init(sc)
df <- createDataFrame(sqlContext, data.frame(a = 1:20))
expect_equal(count(df), 20)
sparkR.stop()
}
})
test_that("rdd GC across sparkR.stop", {
sparkR.stop()
sc <- sparkR.init() # sc should get id 0
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment