Skip to content
Snippets Groups Projects
Commit c0ef3afa authored by Sean Owen's avatar Sean Owen Committed by Patrick Wendell
Browse files

SPARK-1071: Tidy logging strategy and use of log4j 

Prompted by a recent thread on the mailing list, I tried and failed to see if Spark can be made independent of log4j. There are a few cases where control of the underlying logging is pretty useful, and to do that, you have to bind to a specific logger.

Instead I propose some tidying that leaves Spark's use of log4j, but gets rid of warnings and should still enable downstream users to switch. The idea is to pipe everything (except log4j) through SLF4J, and have Spark use SLF4J directly when logging, and where Spark needs to output info (REPL and tests), bind from SLF4J to log4j.

This leaves the same behavior in Spark. It means that downstream users who want to use something except log4j should:

- Exclude dependencies on log4j, slf4j-log4j12 from Spark
- Include dependency on log4j-over-slf4j
- Include dependency on another logger X, and another slf4j-X
- Recreate any log config that Spark does, that is needed, in the other logger's config

That sounds about right.

Here are the key changes:

- Include the jcl-over-slf4j shim everywhere by depending on it in core.
- Exclude dependencies on commons-logging from third-party libraries.
- Include the jul-to-slf4j shim everywhere by depending on it in core.
- Exclude slf4j-* dependencies from third-party libraries to prevent collision or warnings
- Added missing slf4j-log4j12 binding to GraphX, Bagel module tests

And minor/incidental changes:

- Update to SLF4J 1.7.5, which happily matches Hadoop 2’s version and is a recommended update over 1.7.2
- (Remove a duplicate HBase dependency declaration in SparkBuild.scala)
- (Remove a duplicate mockito dependency declaration that was causing warnings and bugging me)

Author: Sean Owen <sowen@cloudera.com>

Closes #570 from srowen/SPARK-1071 and squashes the following commits:

52eac9f [Sean Owen] Add slf4j-over-log4j12 dependency to core (non-test) and remove it from things that depend on core.
77a7fa9 [Sean Owen] SPARK-1071: Tidy logging strategy and use of log4j
parent 437b62fc
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 57 additions and 69 deletions
core/pom.xml
+ 22
− 9
View file @ c0ef3afa
...@@ -39,6 +39,12 @@ ...@@ -39,6 +39,12 @@
<dependency> <dependency>
<groupId>net.java.dev.jets3t</groupId> <groupId>net.java.dev.jets3t</groupId>
<artifactId>jets3t</artifactId> <artifactId>jets3t</artifactId>
<exclusions>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions>
</dependency> </dependency>
<dependency> <dependency>
<groupId>org.apache.avro</groupId> <groupId>org.apache.avro</groupId>
...@@ -68,6 +74,22 @@ ...@@ -68,6 +74,22 @@
<groupId>org.slf4j</groupId> <groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId> <artifactId>slf4j-api</artifactId>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</dependency>
<dependency> <dependency>
<groupId>com.ning</groupId> <groupId>com.ning</groupId>
<artifactId>compress-lzf</artifactId> <artifactId>compress-lzf</artifactId>
...@@ -127,10 +149,6 @@ ...@@ -127,10 +149,6 @@
<groupId>io.netty</groupId> <groupId>io.netty</groupId>
<artifactId>netty-all</artifactId> <artifactId>netty-all</artifactId>
</dependency> </dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</dependency>
<dependency> <dependency>
<groupId>com.clearspring.analytics</groupId> <groupId>com.clearspring.analytics</groupId>
<artifactId>stream</artifactId> <artifactId>stream</artifactId>
...@@ -190,11 +208,6 @@ ...@@ -190,11 +208,6 @@
<artifactId>junit-interface</artifactId> <artifactId>junit-interface</artifactId>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>test</scope>
</dependency>
</dependencies> </dependencies>
<build> <build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
......
examples/pom.xml
+ 4
− 4
View file @ c0ef3afa
...@@ -115,6 +115,10 @@ ...@@ -115,6 +115,10 @@
<groupId>org.jboss.netty</groupId> <groupId>org.jboss.netty</groupId>
<artifactId>netty</artifactId> <artifactId>netty</artifactId>
</exclusion> </exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions> </exclusions>
</dependency> </dependency>
<dependency> <dependency>
...@@ -161,10 +165,6 @@ ...@@ -161,10 +165,6 @@
<groupId>jline</groupId> <groupId>jline</groupId>
<artifactId>jline</artifactId> <artifactId>jline</artifactId>
</exclusion> </exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion> <exclusion>
<groupId>org.apache.cassandra.deps</groupId> <groupId>org.apache.cassandra.deps</groupId>
<artifactId>avro</artifactId> <artifactId>avro</artifactId>
......
external/flume/pom.xml
+ 0
− 5
View file @ c0ef3afa
...@@ -74,11 +74,6 @@ ...@@ -74,11 +74,6 @@
<artifactId>junit-interface</artifactId> <artifactId>junit-interface</artifactId>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>test</scope>
</dependency>
</dependencies> </dependencies>
<build> <build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
......
external/kafka/pom.xml
+ 4
− 5
View file @ c0ef3afa
...@@ -61,6 +61,10 @@ ...@@ -61,6 +61,10 @@
<groupId>net.sf.jopt-simple</groupId> <groupId>net.sf.jopt-simple</groupId>
<artifactId>jopt-simple</artifactId> <artifactId>jopt-simple</artifactId>
</exclusion> </exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
</exclusion>
</exclusions> </exclusions>
</dependency> </dependency>
<dependency> <dependency>
...@@ -78,11 +82,6 @@ ...@@ -78,11 +82,6 @@
<artifactId>junit-interface</artifactId> <artifactId>junit-interface</artifactId>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>test</scope>
</dependency>
</dependencies> </dependencies>
<build> <build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
......
external/mqtt/pom.xml
+ 0
− 5
View file @ c0ef3afa
...@@ -89,11 +89,6 @@ ...@@ -89,11 +89,6 @@
<artifactId>junit-interface</artifactId> <artifactId>junit-interface</artifactId>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>test</scope>
</dependency>
</dependencies> </dependencies>
<build> <build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
......
external/twitter/pom.xml
+ 0
− 5
View file @ c0ef3afa
...@@ -70,11 +70,6 @@ ...@@ -70,11 +70,6 @@
<artifactId>junit-interface</artifactId> <artifactId>junit-interface</artifactId>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>test</scope>
</dependency>
</dependencies> </dependencies>
<build> <build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
......
external/zeromq/pom.xml
+ 0
− 5
View file @ c0ef3afa
...@@ -70,11 +70,6 @@ ...@@ -70,11 +70,6 @@
<artifactId>junit-interface</artifactId> <artifactId>junit-interface</artifactId>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>test</scope>
</dependency>
</dependencies> </dependencies>
<build> <build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
......
pom.xml
+ 16
− 12
View file @ c0ef3afa
...@@ -112,7 +112,7 @@ ...@@ -112,7 +112,7 @@
<mesos.version>0.13.0</mesos.version> <mesos.version>0.13.0</mesos.version>
<akka.group>org.spark-project.akka</akka.group> <akka.group>org.spark-project.akka</akka.group>
<akka.version>2.2.3-shaded-protobuf</akka.version> <akka.version>2.2.3-shaded-protobuf</akka.version>
<slf4j.version>1.7.2</slf4j.version> <slf4j.version>1.7.5</slf4j.version>
<log4j.version>1.2.17</log4j.version> <log4j.version>1.2.17</log4j.version>
<hadoop.version>1.0.4</hadoop.version> <hadoop.version>1.0.4</hadoop.version>
<protobuf.version>2.4.1</protobuf.version> <protobuf.version>2.4.1</protobuf.version>
...@@ -185,6 +185,17 @@ ...@@ -185,6 +185,17 @@
<artifactId>jul-to-slf4j</artifactId> <artifactId>jul-to-slf4j</artifactId>
<version>${slf4j.version}</version> <version>${slf4j.version}</version>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>${slf4j.version}</version>
<!-- <scope>runtime</scope> --> <!-- more correct, but scalac 2.10.3 doesn't like it -->
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency> <dependency>
<groupId>com.ning</groupId> <groupId>com.ning</groupId>
<artifactId>compress-lzf</artifactId> <artifactId>compress-lzf</artifactId>
...@@ -346,23 +357,12 @@ ...@@ -346,23 +357,12 @@
<artifactId>scala-library</artifactId> <artifactId>scala-library</artifactId>
<version>${scala.version}</version> <version>${scala.version}</version>
</dependency> </dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency> <dependency>
<groupId>org.scalatest</groupId> <groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.binary.version}</artifactId> <artifactId>scalatest_${scala.binary.version}</artifactId>
<version>1.9.1</version> <version>1.9.1</version>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<scope>test</scope>
<version>1.8.5</version>
</dependency>
<dependency> <dependency>
<groupId>commons-io</groupId> <groupId>commons-io</groupId>
<artifactId>commons-io</artifactId> <artifactId>commons-io</artifactId>
...@@ -424,6 +424,10 @@ ...@@ -424,6 +424,10 @@
<groupId>org.sonatype.sisu.inject</groupId> <groupId>org.sonatype.sisu.inject</groupId>
<artifactId>*</artifactId> <artifactId>*</artifactId>
</exclusion> </exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
</exclusions> </exclusions>
</dependency> </dependency>
<dependency> <dependency>
......
project/SparkBuild.scala
+ 11
− 10
View file @ c0ef3afa
...@@ -86,7 +86,7 @@ object SparkBuild extends Build { ...@@ -86,7 +86,7 @@ object SparkBuild extends Build {
case None => DEFAULT_YARN case None => DEFAULT_YARN
case Some(v) => v.toBoolean case Some(v) => v.toBoolean
} }
lazy val hadoopClient = if (hadoopVersion.startsWith("0.20.") || hadoopVersion == "1.0.0") "hadoop-core" else "hadoop-client" lazy val hadoopClient = if (hadoopVersion.startsWith("0.20.") || hadoopVersion == "1.0.0") "hadoop-core" else "hadoop-client"
// Conditionally include the yarn sub-project // Conditionally include the yarn sub-project
lazy val yarnAlpha = Project("yarn-alpha", file("yarn/alpha"), settings = yarnAlphaSettings) dependsOn(core) lazy val yarnAlpha = Project("yarn-alpha", file("yarn/alpha"), settings = yarnAlphaSettings) dependsOn(core)
...@@ -236,13 +236,15 @@ object SparkBuild extends Build { ...@@ -236,13 +236,15 @@ object SparkBuild extends Build {
publishLocalBoth <<= Seq(publishLocal in MavenCompile, publishLocal).dependOn publishLocalBoth <<= Seq(publishLocal in MavenCompile, publishLocal).dependOn
) ++ net.virtualvoid.sbt.graph.Plugin.graphSettings ++ ScalaStyleSettings ) ++ net.virtualvoid.sbt.graph.Plugin.graphSettings ++ ScalaStyleSettings
val slf4jVersion = "1.7.2" val slf4jVersion = "1.7.5"
val excludeCglib = ExclusionRule(organization = "org.sonatype.sisu.inject") val excludeCglib = ExclusionRule(organization = "org.sonatype.sisu.inject")
val excludeJackson = ExclusionRule(organization = "org.codehaus.jackson") val excludeJackson = ExclusionRule(organization = "org.codehaus.jackson")
val excludeNetty = ExclusionRule(organization = "org.jboss.netty") val excludeNetty = ExclusionRule(organization = "org.jboss.netty")
val excludeAsm = ExclusionRule(organization = "asm") val excludeAsm = ExclusionRule(organization = "asm")
val excludeSnappy = ExclusionRule(organization = "org.xerial.snappy") val excludeSnappy = ExclusionRule(organization = "org.xerial.snappy")
val excludeCommonsLogging = ExclusionRule(organization = "commons-logging")
val excludeSLF4J = ExclusionRule(organization = "org.slf4j")
def coreSettings = sharedSettings ++ Seq( def coreSettings = sharedSettings ++ Seq(
name := "spark-core", name := "spark-core",
...@@ -257,6 +259,8 @@ object SparkBuild extends Build { ...@@ -257,6 +259,8 @@ object SparkBuild extends Build {
"log4j" % "log4j" % "1.2.17", "log4j" % "log4j" % "1.2.17",
"org.slf4j" % "slf4j-api" % slf4jVersion, "org.slf4j" % "slf4j-api" % slf4jVersion,
"org.slf4j" % "slf4j-log4j12" % slf4jVersion, "org.slf4j" % "slf4j-log4j12" % slf4jVersion,
"org.slf4j" % "jul-to-slf4j" % slf4jVersion,
"org.slf4j" % "jcl-over-slf4j" % slf4jVersion,
"commons-daemon" % "commons-daemon" % "1.0.10", // workaround for bug HADOOP-9407 "commons-daemon" % "commons-daemon" % "1.0.10", // workaround for bug HADOOP-9407
"com.ning" % "compress-lzf" % "1.0.0", "com.ning" % "compress-lzf" % "1.0.0",
"org.xerial.snappy" % "snappy-java" % "1.0.5", "org.xerial.snappy" % "snappy-java" % "1.0.5",
...@@ -268,9 +272,9 @@ object SparkBuild extends Build { ...@@ -268,9 +272,9 @@ object SparkBuild extends Build {
"it.unimi.dsi" % "fastutil" % "6.4.4", "it.unimi.dsi" % "fastutil" % "6.4.4",
"colt" % "colt" % "1.2.0", "colt" % "colt" % "1.2.0",
"org.apache.mesos" % "mesos" % "0.13.0", "org.apache.mesos" % "mesos" % "0.13.0",
"net.java.dev.jets3t" % "jets3t" % "0.7.1", "net.java.dev.jets3t" % "jets3t" % "0.7.1" excludeAll(excludeCommonsLogging),
"org.apache.derby" % "derby" % "10.4.2.0" % "test", "org.apache.derby" % "derby" % "10.4.2.0" % "test",
"org.apache.hadoop" % hadoopClient % hadoopVersion excludeAll(excludeJackson, excludeNetty, excludeAsm, excludeCglib), "org.apache.hadoop" % hadoopClient % hadoopVersion excludeAll(excludeJackson, excludeNetty, excludeAsm, excludeCglib, excludeCommonsLogging, excludeSLF4J),
"org.apache.avro" % "avro" % "1.7.4", "org.apache.avro" % "avro" % "1.7.4",
"org.apache.avro" % "avro-ipc" % "1.7.4" excludeAll(excludeNetty), "org.apache.avro" % "avro-ipc" % "1.7.4" excludeAll(excludeNetty),
"org.apache.zookeeper" % "zookeeper" % "3.4.5" excludeAll(excludeNetty), "org.apache.zookeeper" % "zookeeper" % "3.4.5" excludeAll(excludeNetty),
...@@ -300,18 +304,15 @@ object SparkBuild extends Build { ...@@ -300,18 +304,15 @@ object SparkBuild extends Build {
name := "spark-examples", name := "spark-examples",
libraryDependencies ++= Seq( libraryDependencies ++= Seq(
"com.twitter" %% "algebird-core" % "0.1.11", "com.twitter" %% "algebird-core" % "0.1.11",
"org.apache.hbase" % "hbase" % "0.94.6" excludeAll(excludeNetty, excludeAsm), "org.apache.hbase" % "hbase" % HBASE_VERSION excludeAll(excludeNetty, excludeAsm, excludeCommonsLogging),
"org.apache.hbase" % "hbase" % HBASE_VERSION excludeAll(excludeNetty, excludeAsm),
"org.apache.cassandra" % "cassandra-all" % "1.2.6" "org.apache.cassandra" % "cassandra-all" % "1.2.6"
exclude("com.google.guava", "guava") exclude("com.google.guava", "guava")
exclude("com.googlecode.concurrentlinkedhashmap", "concurrentlinkedhashmap-lru") exclude("com.googlecode.concurrentlinkedhashmap", "concurrentlinkedhashmap-lru")
exclude("com.ning","compress-lzf") exclude("com.ning","compress-lzf")
exclude("io.netty", "netty") exclude("io.netty", "netty")
exclude("jline","jline") exclude("jline","jline")
exclude("log4j","log4j")
exclude("org.apache.cassandra.deps", "avro") exclude("org.apache.cassandra.deps", "avro")
excludeAll(excludeSnappy) excludeAll(excludeSnappy, excludeCglib, excludeSLF4J)
excludeAll(excludeCglib)
) )
) ++ assemblySettings ++ extraAssemblySettings ) ++ assemblySettings ++ extraAssemblySettings
...@@ -416,7 +417,7 @@ object SparkBuild extends Build { ...@@ -416,7 +417,7 @@ object SparkBuild extends Build {
exclude("com.sun.jdmk", "jmxtools") exclude("com.sun.jdmk", "jmxtools")
exclude("com.sun.jmx", "jmxri") exclude("com.sun.jmx", "jmxri")
exclude("net.sf.jopt-simple", "jopt-simple") exclude("net.sf.jopt-simple", "jopt-simple")
excludeAll(excludeNetty) excludeAll(excludeNetty, excludeSLF4J)
) )
) )
......
repl/pom.xml
+ 0
− 4
View file @ c0ef3afa
...@@ -72,10 +72,6 @@ ...@@ -72,10 +72,6 @@
<groupId>org.slf4j</groupId> <groupId>org.slf4j</groupId>
<artifactId>jul-to-slf4j</artifactId> <artifactId>jul-to-slf4j</artifactId>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</dependency>
<dependency> <dependency>
<groupId>org.scalatest</groupId> <groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.binary.version}</artifactId> <artifactId>scalatest_${scala.binary.version}</artifactId>
......
streaming/pom.xml
+ 0
− 5
View file @ c0ef3afa
...@@ -79,11 +79,6 @@ ...@@ -79,11 +79,6 @@
<artifactId>junit-interface</artifactId> <artifactId>junit-interface</artifactId>
<scope>test</scope> <scope>test</scope>
</dependency> </dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>test</scope>
</dependency>
<dependency> <dependency>
<groupId>commons-io</groupId> <groupId>commons-io</groupId>
<artifactId>commons-io</artifactId> <artifactId>commons-io</artifactId>
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment