-
- Downloads
[SPARK-12362][SQL][WIP] Inline Hive Parser
This is a WIP. The PR has been taken over from nongli (see https://github.com/apache/spark/pull/10420). I have removed some additional dead code, and fixed a few issues which were caused by the fact that the inlined Hive parser is newer than the Hive parser we currently use in Spark. I am submitting this PR in order to get some feedback and testing done. There is quite a bit of work to do: - [ ] Get it to pass jenkins build/test. - [ ] Aknowledge Hive-project for using their parser. - [ ] Refactorings between HiveQl and the java classes. - [ ] Create our own ASTNode and integrate the current implicit extentions. - [ ] Move remaining ```SemanticAnalyzer``` and ```ParseUtils``` functionality to ```HiveQl```. - [ ] Removing Hive dependencies from the parser. This will require some edits in the grammar files. - [ ] Introduce our own context which needs to contain a ```TokenRewriteStream```. - [ ] Add ```useSQL11ReservedKeywordsForIdentifier``` and ```allowQuotedId``` to the catalyst or sql configuration. - [ ] Remove ```HiveConf``` from grammar files &HiveQl, and pass in our own configuration. - [ ] Moving the parser into sql/core. cc nongli rxin Author: Herman van Hovell <hvanhovell@questtec.nl> Author: Nong Li <nong@databricks.com> Author: Nong Li <nongli@gmail.com> Closes #10509 from hvanhovell/SPARK-12362.
Showing
- pom.xml 5 additions, 0 deletionspom.xml
- project/SparkBuild.scala 1 addition, 1 deletionproject/SparkBuild.scala
- project/plugins.sbt 4 additions, 0 deletionsproject/plugins.sbt
- sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala 6 additions, 4 deletions...che/spark/sql/hive/execution/HiveCompatibilitySuite.scala
- sql/hive/pom.xml 22 additions, 0 deletionssql/hive/pom.xml
- sql/hive/src/main/antlr3/org/apache/spark/sql/parser/FromClauseParser.g 330 additions, 0 deletions...ain/antlr3/org/apache/spark/sql/parser/FromClauseParser.g
- sql/hive/src/main/antlr3/org/apache/spark/sql/parser/IdentifiersParser.g 697 additions, 0 deletions...in/antlr3/org/apache/spark/sql/parser/IdentifiersParser.g
- sql/hive/src/main/antlr3/org/apache/spark/sql/parser/SelectClauseParser.g 226 additions, 0 deletions...n/antlr3/org/apache/spark/sql/parser/SelectClauseParser.g
- sql/hive/src/main/antlr3/org/apache/spark/sql/parser/SparkSqlLexer.g 474 additions, 0 deletions...c/main/antlr3/org/apache/spark/sql/parser/SparkSqlLexer.g
- sql/hive/src/main/antlr3/org/apache/spark/sql/parser/SparkSqlParser.g 2457 additions, 0 deletions.../main/antlr3/org/apache/spark/sql/parser/SparkSqlParser.g
- sql/hive/src/main/java/org/apache/spark/sql/parser/ASTErrorNode.java 49 additions, 0 deletions...c/main/java/org/apache/spark/sql/parser/ASTErrorNode.java
- sql/hive/src/main/java/org/apache/spark/sql/parser/ASTNode.java 245 additions, 0 deletions...ve/src/main/java/org/apache/spark/sql/parser/ASTNode.java
- sql/hive/src/main/java/org/apache/spark/sql/parser/ParseDriver.java 213 additions, 0 deletions...rc/main/java/org/apache/spark/sql/parser/ParseDriver.java
- sql/hive/src/main/java/org/apache/spark/sql/parser/ParseError.java 54 additions, 0 deletions...src/main/java/org/apache/spark/sql/parser/ParseError.java
- sql/hive/src/main/java/org/apache/spark/sql/parser/ParseException.java 51 additions, 0 deletions...main/java/org/apache/spark/sql/parser/ParseException.java
- sql/hive/src/main/java/org/apache/spark/sql/parser/ParseUtils.java 96 additions, 0 deletions...src/main/java/org/apache/spark/sql/parser/ParseUtils.java
- sql/hive/src/main/java/org/apache/spark/sql/parser/SemanticAnalyzer.java 406 additions, 0 deletions...in/java/org/apache/spark/sql/parser/SemanticAnalyzer.java
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 66 additions, 67 deletions...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
Loading
Please register or sign in to comment