Skip to content
Snippets Groups Projects
Commit 69d1c4c5 authored by Dongjoon Hyun's avatar Dongjoon Hyun Committed by Shixiong Zhu
Browse files

[SPARK-19137][SQL] Fix `withSQLConf` to reset `OptionalConfigEntry` correctly


## What changes were proposed in this pull request?

`DataStreamReaderWriterSuite` makes test files in source folder like the followings. Interestingly, the root cause is `withSQLConf` fails to reset `OptionalConfigEntry` correctly. In other words, it resets the config into `Some(undefined)`.

```bash
$ git status
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        sql/core/%253Cundefined%253E/
        sql/core/%3Cundefined%3E/
```

## How was this patch tested?

Manual.
```
build/sbt "project sql" test
git status
```

Author: Dongjoon Hyun <dongjoon@apache.org>

Closes #16522 from dongjoon-hyun/SPARK-19137.

(cherry picked from commit d5b1dc93)
Signed-off-by: default avatarShixiong Zhu <shixiong@databricks.com>
parent 65c866ef
No related branches found
No related tags found
No related merge requests found
......@@ -94,7 +94,13 @@ private[sql] trait SQLTestUtils
*/
protected def withSQLConf(pairs: (String, String)*)(f: => Unit): Unit = {
val (keys, values) = pairs.unzip
val currentValues = keys.map(key => Try(spark.conf.get(key)).toOption)
val currentValues = keys.map { key =>
if (spark.conf.contains(key)) {
Some(spark.conf.get(key))
} else {
None
}
}
(keys, values).zipped.foreach(spark.conf.set)
try f finally {
keys.zip(currentValues).foreach {
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment