-
- Downloads
[SPARK-7133] [SQL] Implement struct, array, and map field accessor
It's the first step: generalize UnresolvedGetField to support all map, struct, and array TODO: add `apply` in Scala and `__getitem__` in Python, and unify the `getItem` and `getField` methods to one single API(or should we keep them for compatibility?). Author: Wenchen Fan <cloud0fan@outlook.com> Closes #5744 from cloud-fan/generalize and squashes the following commits: 715c589 [Wenchen Fan] address comments 7ea5b31 [Wenchen Fan] fix python test 4f0833a [Wenchen Fan] add python test f515d69 [Wenchen Fan] add apply method and test cases 8df6199 [Wenchen Fan] fix python test 239730c [Wenchen Fan] fix test compile 2a70526 [Wenchen Fan] use _bin_op in dataframe.py 6bf72bc [Wenchen Fan] address comments 3f880c3 [Wenchen Fan] add java doc ab35ab5 [Wenchen Fan] fix python test b5961a9 [Wenchen Fan] fix style c9d85f5 [Wenchen Fan] generalize UnresolvedGetField to support all map, struct, and array
Showing
- python/pyspark/sql/dataframe.py 12 additions, 12 deletionspython/pyspark/sql/dataframe.py
- python/pyspark/sql/tests.py 7 additions, 0 deletionspython/pyspark/sql/tests.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala 2 additions, 2 deletions.../main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 2 additions, 2 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala 12 additions, 2 deletions...a/org/apache/spark/sql/catalyst/analysis/unresolved.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/dsl/package.scala 4 additions, 3 deletions...ain/scala/org/apache/spark/sql/catalyst/dsl/package.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExtractValue.scala 206 additions, 0 deletions.../apache/spark/sql/catalyst/expressions/ExtractValue.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypes.scala 0 additions, 131 deletions.../apache/spark/sql/catalyst/expressions/complexTypes.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala 2 additions, 4 deletions...a/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala 1 addition, 1 deletion...ala/org/apache/spark/sql/catalyst/planning/patterns.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala 2 additions, 1 deletion...apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ExpressionEvaluationSuite.scala 47 additions, 22 deletions.../sql/catalyst/expressions/ExpressionEvaluationSuite.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala 4 additions, 4 deletions...e/spark/sql/catalyst/optimizer/ConstantFoldingSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/Column.scala 16 additions, 3 deletionssql/core/src/main/scala/org/apache/spark/sql/Column.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 8 additions, 2 deletions.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala 2 additions, 2 deletions...ive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala
Loading
Please register or sign in to comment