Skip to content
  • Nong Li's avatar
    b600bccf
    [SPARK-12362][SQL][WIP] Inline Hive Parser · b600bccf
    Nong Li authored
    This is a WIP. The PR has been taken over from nongli (see https://github.com/apache/spark/pull/10420). I have removed some additional dead code, and fixed a few issues which were caused by the fact that the inlined Hive parser is newer than the Hive parser we currently use in Spark.
    
    I am submitting this PR in order to get some feedback and testing done. There is quite a bit of work to do:
    - [ ] Get it to pass jenkins build/test.
    - [ ] Aknowledge Hive-project for using their parser.
    - [ ] Refactorings between HiveQl and the java classes.
      - [ ] Create our own ASTNode and integrate the current implicit extentions.
      - [ ] Move remaining ```SemanticAnalyzer``` and ```ParseUtils``` functionality to ```HiveQl```.
    - [ ] Removing Hive dependencies from the parser. This will require some edits in the grammar files.
      - [ ] Introduce our own context which needs to contain a ```TokenRewriteStream```.
      - [ ] Add ```useSQL11ReservedKeywordsForIdentifier``` and ```allowQuotedId``` to the catalyst or sql configuration.
      - [ ] Remove ```HiveConf``` from grammar files &HiveQl, and pass in our own configuration.
    - [ ] Moving the parser into sql/core.
    
    cc nongli rxin
    
    Author: Herman van Hovell <hvanhovell@questtec.nl>
    Author: Nong Li <nong@databricks.com>
    Author: Nong Li <nongli@gmail.com>
    
    Closes #10509 from hvanhovell/SPARK-12362.
    b600bccf
    [SPARK-12362][SQL][WIP] Inline Hive Parser
    Nong Li authored
    This is a WIP. The PR has been taken over from nongli (see https://github.com/apache/spark/pull/10420). I have removed some additional dead code, and fixed a few issues which were caused by the fact that the inlined Hive parser is newer than the Hive parser we currently use in Spark.
    
    I am submitting this PR in order to get some feedback and testing done. There is quite a bit of work to do:
    - [ ] Get it to pass jenkins build/test.
    - [ ] Aknowledge Hive-project for using their parser.
    - [ ] Refactorings between HiveQl and the java classes.
      - [ ] Create our own ASTNode and integrate the current implicit extentions.
      - [ ] Move remaining ```SemanticAnalyzer``` and ```ParseUtils``` functionality to ```HiveQl```.
    - [ ] Removing Hive dependencies from the parser. This will require some edits in the grammar files.
      - [ ] Introduce our own context which needs to contain a ```TokenRewriteStream```.
      - [ ] Add ```useSQL11ReservedKeywordsForIdentifier``` and ```allowQuotedId``` to the catalyst or sql configuration.
      - [ ] Remove ```HiveConf``` from grammar files &HiveQl, and pass in our own configuration.
    - [ ] Moving the parser into sql/core.
    
    cc nongli rxin
    
    Author: Herman van Hovell <hvanhovell@questtec.nl>
    Author: Nong Li <nong@databricks.com>
    Author: Nong Li <nongli@gmail.com>
    
    Closes #10509 from hvanhovell/SPARK-12362.
Loading