Skip to content
Snippets Groups Projects
Commit 37482ce5 authored by Jim Carroll's avatar Jim Carroll Committed by Michael Armbrust
Browse files

[SPARK-4412][SQL] Fix Spark's control of Parquet logging.

The Spark ParquetRelation.scala code makes the assumption that the parquet.Log class has already been loaded. If ParquetRelation.enableLogForwarding executes prior to the parquet.Log class being loaded then the code in enableLogForwarding has no affect.

ParquetRelation.scala attempts to override the parquet logger but, at least currently (and if your application simply reads a parquet file before it does anything else with Parquet), the parquet.Log class hasn't been loaded yet. Therefore the code in ParquetRelation.enableLogForwarding has no affect. If you look at the code in parquet.Log there's a static initializer that needs to be called prior to enableLogForwarding or whatever enableLogForwarding does gets undone by this static initializer.

The "fix" would be to force the static initializer to get called in parquet.Log as part of enableForwardLogging.

Author: Jim Carroll <jim@dontcallme.com>

Closes #3271 from jimfcarroll/parquet-logging and squashes the following commits:

37bdff7 [Jim Carroll] Fix Spark's control of Parquet logging.
parent 63ca3af6
No related branches found
No related tags found
No related merge requests found
......@@ -84,6 +84,21 @@ private[sql] case class ParquetRelation(
private[sql] object ParquetRelation {
def enableLogForwarding() {
// Note: the parquet.Log class has a static initializer that
// sets the java.util.logging Logger for "parquet". This
// checks first to see if there's any handlers already set
// and if not it creates them. If this method executes prior
// to that class being loaded then:
// 1) there's no handlers installed so there's none to
// remove. But when it IS finally loaded the desired affect
// of removing them is circumvented.
// 2) The parquet.Log static initializer calls setUseParentHanders(false)
// undoing the attempt to override the logging here.
//
// Therefore we need to force the class to be loaded.
// This should really be resolved by Parquet.
Class.forName(classOf[parquet.Log].getName())
// Note: Logger.getLogger("parquet") has a default logger
// that appends to Console which needs to be cleared.
val parquetLogger = java.util.logging.Logger.getLogger("parquet")
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment