-
- Downloads
[SPARK-13681][SPARK-14458][SPARK-14566][SQL] Add back once removed...
[SPARK-13681][SPARK-14458][SPARK-14566][SQL] Add back once removed CommitFailureTestRelationSuite and SimpleTextHadoopFsRelationSuite ## What changes were proposed in this pull request? These test suites were removed while refactoring `HadoopFsRelation` related API. This PR brings them back. This PR also fixes two regressions: - SPARK-14458, which causes runtime error when saving partitioned tables using `FileFormat` data sources that are not able to infer their own schemata. This bug wasn't detected by any built-in data sources because all of them happen to have schema inference feature. - SPARK-14566, which happens to be covered by SPARK-14458 and causes wrong query result or runtime error when - appending a Dataset `ds` to a persisted partitioned data source relation `t`, and - partition columns in `ds` don't all appear after data columns ## How was this patch tested? `CommitFailureTestRelationSuite` uses a testing relation that always fails when committing write tasks to test write job cleanup. `SimpleTextHadoopFsRelationSuite` uses a testing relation to test general `HadoopFsRelation` and `FileFormat` interfaces. The two regressions are both covered by existing test cases. Author: Cheng Lian <lian@databricks.com> Closes #12179 from liancheng/spark-13681-commit-failure-test.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala 12 additions, 2 deletions...g/apache/spark/sql/execution/datasources/DataSource.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala 6 additions, 5 deletions...cala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala 1 addition, 1 deletion.../scala/org/apache/spark/sql/hive/execution/commands.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/CommitFailureTestRelationSuite.scala 82 additions, 0 deletions...he/spark/sql/sources/CommitFailureTestRelationSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/CommitFailureTestSource.scala 67 additions, 0 deletions...rg/apache/spark/sql/sources/CommitFailureTestSource.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextHadoopFsRelationSuite.scala 68 additions, 0 deletions...e/spark/sql/sources/SimpleTextHadoopFsRelationSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextRelation.scala 167 additions, 0 deletions...ala/org/apache/spark/sql/sources/SimpleTextRelation.scala
Loading
Please register or sign in to comment