-
- Downloads
[SQL] Improve DataFrame API error reporting
1. Throw UnsupportedOperationException if a Column is not computable. 2. Perform eager analysis on DataFrame so we can catch errors when they happen (not when an action is run). Author: Reynold Xin <rxin@databricks.com> Author: Davies Liu <davies@databricks.com> Closes #4296 from rxin/col-computability and squashes the following commits: 6527b86 [Reynold Xin] Merge pull request #8 from davies/col-computability fd92bc7 [Reynold Xin] Merge branch 'master' into col-computability f79034c [Davies Liu] fix python tests 5afe1ff [Reynold Xin] Fix scala test. 17f6bae [Reynold Xin] Various fixes. b932e86 [Reynold Xin] Added eager analysis for error reporting. e6f00b8 [Reynold Xin] [SQL][API] ComputableColumn vs IncomputableColumn
Showing
- python/pyspark/sql.py 52 additions, 23 deletionspython/pyspark/sql.py
- python/pyspark/tests.py 4 additions, 2 deletionspython/pyspark/tests.py
- sql/catalyst/src/main/scala/org/apache/spark/sql/types/dataTypes.scala 2 additions, 1 deletion...src/main/scala/org/apache/spark/sql/types/dataTypes.scala
- sql/core/src/main/scala/org/apache/spark/sql/Column.scala 147 additions, 94 deletionssql/core/src/main/scala/org/apache/spark/sql/Column.scala
- sql/core/src/main/scala/org/apache/spark/sql/ComputableColumn.scala 33 additions, 0 deletions...rc/main/scala/org/apache/spark/sql/ComputableColumn.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 68 additions, 224 deletionssql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameImpl.scala 331 additions, 0 deletions...e/src/main/scala/org/apache/spark/sql/DataFrameImpl.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dsl.scala 15 additions, 6 deletionssql/core/src/main/scala/org/apache/spark/sql/Dsl.scala
- sql/core/src/main/scala/org/apache/spark/sql/GroupedDataFrame.scala 6 additions, 5 deletions...rc/main/scala/org/apache/spark/sql/GroupedDataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/IncomputableColumn.scala 160 additions, 0 deletions.../main/scala/org/apache/spark/sql/IncomputableColumn.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 9 additions, 9 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala 1 addition, 1 deletion.../main/scala/org/apache/spark/sql/execution/commands.scala
- sql/core/src/main/scala/org/apache/spark/sql/sources/ddl.scala 1 addition, 1 deletion...ore/src/main/scala/org/apache/spark/sql/sources/ddl.scala
- sql/core/src/main/scala/org/apache/spark/sql/test/TestSQLContext.scala 1 addition, 1 deletion...main/scala/org/apache/spark/sql/test/TestSQLContext.scala
- sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala 39 additions, 0 deletions...st/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 13 additions, 0 deletions.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 2 additions, 4 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetFilterSuite.scala 3 additions, 3 deletions...ala/org/apache/spark/sql/parquet/ParquetFilterSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala 2 additions, 1 deletion...rc/main/scala/org/apache/spark/sql/hive/HiveContext.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 7 additions, 6 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
Loading
Please register or sign in to comment