Skip to content
Snippets Groups Projects
Commit b3a07526 authored by Sean Owen's avatar Sean Owen Committed by Herman van Hovell
Browse files

[SPARK-21718][SQL] Heavy log of type: "Skipping partition based on stats ..."

## What changes were proposed in this pull request?

Reduce 'Skipping partitions' message to debug

## How was this patch tested?

Existing tests

Author: Sean Owen <sowen@cloudera.com>

Closes #19010 from srowen/SPARK-21718.
parent 77d046ec
No related branches found
No related tags found
No related merge requests found
......@@ -166,12 +166,13 @@ case class InMemoryTableScanExec(
if (inMemoryPartitionPruningEnabled) {
cachedBatchIterator.filter { cachedBatch =>
if (!partitionFilter.eval(cachedBatch.stats)) {
def statsString: String = schemaIndex.map {
case (a, i) =>
logDebug {
val statsString = schemaIndex.map { case (a, i) =>
val value = cachedBatch.stats.get(i, a.dataType)
s"${a.name}: $value"
}.mkString(", ")
logInfo(s"Skipping partition based on stats $statsString")
}.mkString(", ")
s"Skipping partition based on stats $statsString"
}
false
} else {
true
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment