-
- Downloads
[SPARK-13440][SQL] ObjectType should accept any ObjectType, If should not care about nullability
The type checking functions of `If` and `UnwrapOption` are fixed to eliminate spurious failures. `UnwrapOption` was checking for an input of `ObjectType` but `ObjectType`'s accept function was hard coded to return `false`. `If`'s type check was returning a false negative in the case that the two options differed only by nullability. Tests added: - an end-to-end regression test is added to `DatasetSuite` for the reported failure. - all the unit tests in `ExpressionEncoderSuite` are augmented to also confirm successful analysis. These tests are actually what pointed out the additional issues with `If` resolution. Author: Michael Armbrust <michael@databricks.com> Closes #11316 from marmbrus/datasetOptions.
Showing
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/conditionalExpressions.scala 1 addition, 1 deletion...ark/sql/catalyst/expressions/conditionalExpressions.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LocalRelation.scala 3 additions, 0 deletions...ache/spark/sql/catalyst/plans/logical/LocalRelation.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/ObjectType.scala 4 additions, 2 deletions...rc/main/scala/org/apache/spark/sql/types/ObjectType.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisTest.scala 12 additions, 1 deletion...org/apache/spark/sql/catalyst/analysis/AnalysisTest.scala
- sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala 15 additions, 4 deletions.../spark/sql/catalyst/encoders/ExpressionEncoderSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 8 additions, 0 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Loading
Please register or sign in to comment