-
- Downloads
[SPARK-20399][SQL] Add a config to fallback string literal parsing consistent...
[SPARK-20399][SQL] Add a config to fallback string literal parsing consistent with old sql parser behavior ## What changes were proposed in this pull request? The new SQL parser is introduced into Spark 2.0. All string literals are unescaped in parser. Seems it bring an issue regarding the regex pattern string. The following codes can reproduce it: val data = Seq("\u0020\u0021\u0023", "abc") val df = data.toDF() // 1st usage: works in 1.6 // Let parser parse pattern string val rlike1 = df.filter("value rlike '^\\x20[\\x20-\\x23]+$'") // 2nd usage: works in 1.6, 2.x // Call Column.rlike so the pattern string is a literal which doesn't go through parser val rlike2 = df.filter($"value".rlike("^\\x20[\\x20-\\x23]+$")) // In 2.x, we need add backslashes to make regex pattern parsed correctly val rlike3 = df.filter("value rlike '^\\\\x20[\\\\x20-\\\\x23]+$'") Follow the discussion in #17736, this patch adds a config to fallback to 1.6 string literal parsing and mitigate migration issue. ## How was this patch tested? Jenkins tests. Please review http://spark.apache.org/contributing.html before opening a pull request. Author: Liang-Chi Hsieh <viirya@gmail.com> Closes #17887 from viirya/add-config-fallback-string-parsing.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala 1 addition, 1 deletion...rg/apache/spark/sql/catalyst/catalog/SessionCatalog.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala 32 additions, 1 deletion...he/spark/sql/catalyst/expressions/regexpExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala 9 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseDriver.scala 7 additions, 1 deletion...la/org/apache/spark/sql/catalyst/parser/ParseDriver.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/ParserUtils.scala 6 additions, 0 deletions...la/org/apache/spark/sql/catalyst/parser/ParserUtils.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 10 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/internal/SQLConf.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala 92 additions, 36 deletions...che/spark/sql/catalyst/parser/ExpressionParserSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala 1 addition, 1 deletion...scala/org/apache/spark/sql/execution/SparkSqlParser.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 13 additions, 0 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Loading
Please register or sign in to comment