-
- Downloads
[SPARK-12799] Simplify various string output for expressions
This PR introduces several major changes: 1. Replacing `Expression.prettyString` with `Expression.sql` The `prettyString` method is mostly an internal, developer faced facility for debugging purposes, and shouldn't be exposed to users. 1. Using SQL-like representation as column names for selected fields that are not named expression (back-ticks and double quotes should be removed) Before, we were using `prettyString` as column names when possible, and sometimes the result column names can be weird. Here are several examples: Expression | `prettyString` | `sql` | Note ------------------ | -------------- | ---------- | --------------- `a && b` | `a && b` | `a AND b` | `a.getField("f")` | `a[f]` | `a.f` | `a` is a struct 1. Adding trait `NonSQLExpression` extending from `Expression` for expressions that don't have a SQL representation (e.g. Scala UDF/UDAF and Java/Scala object expressions used for encoders) `NonSQLExpression.sql` may return an arbitrary user facing string representation of the expression. Author: Cheng Lian <lian@databricks.com> Closes #10757 from liancheng/spark-12799.simplify-expression-string-methods.
Showing
- R/pkg/inst/tests/testthat/test_sparkSQL.R 2 additions, 2 deletionsR/pkg/inst/tests/testthat/test_sparkSQL.R
- python/pyspark/sql/column.py 16 additions, 16 deletionspython/pyspark/sql/column.py
- python/pyspark/sql/context.py 5 additions, 5 deletionspython/pyspark/sql/context.py
- python/pyspark/sql/functions.py 15 additions, 15 deletionspython/pyspark/sql/functions.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala 5 additions, 3 deletions...ala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala 13 additions, 13 deletions...rg/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DistinctAggregationRewriter.scala 3 additions, 3 deletions...k/sql/catalyst/analysis/DistinctAggregationRewriter.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala 9 additions, 8 deletions...a/org/apache/spark/sql/catalyst/analysis/unresolved.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExpectsInputTypes.scala 1 addition, 1 deletion...he/spark/sql/catalyst/expressions/ExpectsInputTypes.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala 29 additions, 42 deletions...rg/apache/spark/sql/catalyst/expressions/Expression.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala 2 additions, 2 deletions.../org/apache/spark/sql/catalyst/expressions/ScalaUDF.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala 1 addition, 4 deletions...spark/sql/catalyst/expressions/aggregate/interfaces.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala 6 additions, 2 deletions...rg/apache/spark/sql/catalyst/expressions/arithmetic.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitwiseExpressions.scala 2 additions, 0 deletions...e/spark/sql/catalyst/expressions/bitwiseExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodegenFallback.scala 3 additions, 2 deletions...rk/sql/catalyst/expressions/codegen/CodegenFallback.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala 14 additions, 8 deletions...park/sql/catalyst/expressions/complexTypeExtractors.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala 1 addition, 1 deletion...ark/sql/catalyst/expressions/conditionalExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala 11 additions, 28 deletions.../org/apache/spark/sql/catalyst/expressions/literals.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala 4 additions, 2 deletions...ache/spark/sql/catalyst/expressions/mathExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala 15 additions, 5 deletions...che/spark/sql/catalyst/expressions/namedExpressions.scala
Loading
Please register or sign in to comment