Skip to content
Snippets Groups Projects
Commit b3a07526 authored by Sean Owen's avatar Sean Owen Committed by Herman van Hovell
Browse files

[SPARK-21718][SQL] Heavy log of type: "Skipping partition based on stats ..."

## What changes were proposed in this pull request?

Reduce 'Skipping partitions' message to debug

## How was this patch tested?

Existing tests

Author: Sean Owen <sowen@cloudera.com>

Closes #19010 from srowen/SPARK-21718.
parent 77d046ec
No related branches found
No related tags found
No related merge requests found
...@@ -166,12 +166,13 @@ case class InMemoryTableScanExec( ...@@ -166,12 +166,13 @@ case class InMemoryTableScanExec(
if (inMemoryPartitionPruningEnabled) { if (inMemoryPartitionPruningEnabled) {
cachedBatchIterator.filter { cachedBatch => cachedBatchIterator.filter { cachedBatch =>
if (!partitionFilter.eval(cachedBatch.stats)) { if (!partitionFilter.eval(cachedBatch.stats)) {
def statsString: String = schemaIndex.map { logDebug {
case (a, i) => val statsString = schemaIndex.map { case (a, i) =>
val value = cachedBatch.stats.get(i, a.dataType) val value = cachedBatch.stats.get(i, a.dataType)
s"${a.name}: $value" s"${a.name}: $value"
}.mkString(", ") }.mkString(", ")
logInfo(s"Skipping partition based on stats $statsString") s"Skipping partition based on stats $statsString"
}
false false
} else { } else {
true true
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment