Skip to content
Snippets Groups Projects
Commit 3c471885 authored by Yin Huai's avatar Yin Huai
Browse files

[SPARK-11434][SPARK-11103][SQL] Fix test ": Filter applied on merged Parquet...

[SPARK-11434][SPARK-11103][SQL] Fix test ": Filter applied on merged Parquet schema with new column fails"

https://issues.apache.org/jira/browse/SPARK-11434

Author: Yin Huai <yhuai@databricks.com>

Closes #9387 from yhuai/SPARK-11434.
parent 69b9e4b3
No related branches found
No related tags found
No related merge requests found
......@@ -323,15 +323,15 @@ class ParquetFilterSuite extends QueryTest with ParquetTest with SharedSQLContex
withSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true",
SQLConf.PARQUET_SCHEMA_MERGING_ENABLED.key -> "true") {
withTempPath { dir =>
var pathOne = s"${dir.getCanonicalPath}/table1"
val pathOne = s"${dir.getCanonicalPath}/table1"
(1 to 3).map(i => (i, i.toString)).toDF("a", "b").write.parquet(pathOne)
var pathTwo = s"${dir.getCanonicalPath}/table2"
val pathTwo = s"${dir.getCanonicalPath}/table2"
(1 to 3).map(i => (i, i.toString)).toDF("c", "b").write.parquet(pathTwo)
// If the "c = 1" filter gets pushed down, this query will throw an exception which
// Parquet emits. This is a Parquet issue (PARQUET-389).
checkAnswer(
sqlContext.read.parquet(pathOne, pathTwo).filter("c = 1"),
sqlContext.read.parquet(pathOne, pathTwo).filter("c = 1").selectExpr("c", "b", "a"),
(1 to 1).map(i => Row(i, i.toString, null)))
}
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment