Skip to content
Snippets Groups Projects
Commit 9565c246 authored by hyukjinkwon's avatar hyukjinkwon Committed by Michael Armbrust
Browse files

[SPARK-9557][SQL] Refactor ParquetFilterSuite and remove old ParquetFilters code

Actually this was resolved by https://github.com/apache/spark/pull/8275.

But I found the JIRA issue for this is not marked as resolved since the PR above was made for another issue but the PR above resolved both.

I commented that this is resolved by the PR above; however, I opened this PR as I would like to just add
a little bit of corrections.

In the previous PR, I refactored the test by not reducing just collecting filters; however, this would not test  properly `And` filter (which is not given to the tests). I unintentionally changed this from the original way (before being refactored).

In this PR, I just followed the original way to collect filters by reducing.

I would like to close this if this PR is inappropriate and somebody would like this deal with it in the separate PR related with this.

Author: hyukjinkwon <gurwls223@gmail.com>

Closes #9554 from HyukjinKwon/SPARK-9557.
parent fcb57e9c
No related branches found
No related tags found
No related merge requests found
......@@ -54,12 +54,12 @@ class ParquetFilterSuite extends QueryTest with ParquetTest with SharedSQLContex
.select(output.map(e => Column(e)): _*)
.where(Column(predicate))
val analyzedPredicate = query.queryExecution.optimizedPlan.collect {
val maybeAnalyzedPredicate = query.queryExecution.optimizedPlan.collect {
case PhysicalOperation(_, filters, LogicalRelation(_: ParquetRelation, _)) => filters
}.flatten
assert(analyzedPredicate.nonEmpty)
}.flatten.reduceLeftOption(_ && _)
assert(maybeAnalyzedPredicate.isDefined)
val selectedFilters = analyzedPredicate.flatMap(DataSourceStrategy.translateFilter)
val selectedFilters = maybeAnalyzedPredicate.flatMap(DataSourceStrategy.translateFilter)
assert(selectedFilters.nonEmpty)
selectedFilters.foreach { pred =>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment