Skip to content
Snippets Groups Projects
  1. Jan 07, 2014
  2. Jan 06, 2014
    • Tathagata Das's avatar
      Merge remote-tracking branch 'apache/master' into project-refactor · 3b4c4c7f
      Tathagata Das authored
      Conflicts:
      	examples/src/main/java/org/apache/spark/streaming/examples/JavaFlumeEventCount.java
      	streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
      	streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala
      	streaming/src/test/java/org/apache/spark/streaming/JavaAPISuite.java
      	streaming/src/test/scala/org/apache/spark/streaming/InputStreamsSuite.scala
      	streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala
      3b4c4c7f
    • Tathagata Das's avatar
      Changed JavaStreamingContextWith*** to ***Function in streaming.api.java.***... · d0fd3b9a
      Tathagata Das authored
      Changed JavaStreamingContextWith*** to ***Function in streaming.api.java.*** package. Also fixed packages of Flume and MQTT tests.
      d0fd3b9a
    • Patrick Wendell's avatar
      Merge pull request #333 from pwendell/logging-silence · a2e7e049
      Patrick Wendell authored
      Quiet ERROR-level Akka Logs
      
      This fixes an issue I've seen where akka logs a bunch of things at ERROR level when connecting to a standalone cluster, even in the normal case. I noticed that even when lifecycle logging was disabled, the netty code inside of akka still logged away via akka's EndpointWriter class. There are also some other log streams that I think are new in akka 2.2.1 that I've disabled.
      
      Finally, I added some better logging to the standalone client. This makes it more clear when a connection failure occurs what is going on. Previously it never explicitly said if a connection attempt had failed.
      
      The commit messages here have some more detail.
      a2e7e049
  3. Jan 05, 2014
    • Patrick Wendell's avatar
      Responding to Aaron's review · 675d7eb4
      Patrick Wendell authored
      675d7eb4
    • Reynold Xin's avatar
      Merge pull request #334 from pwendell/examples-fix · 5b0986a1
      Reynold Xin authored
      Removing SPARK_EXAMPLES_JAR in the code
      
      This re-writes all of the examples to use the `SparkContext.jarOfClass` mechanism for loading the examples jar. This necessary for environments like YARN and the Standalone mode where example programs will be submit from inside the cluster rather than at the client using `./spark-example`.
      
      This still leaves SPARK_EXAMPLES_JAR in place in the shell scripts for setting up the classpath if `./spark-example` is run.
      5b0986a1
    • Reynold Xin's avatar
      Merge pull request #335 from rxin/ser · f4b924f6
      Reynold Xin authored
      Fall back to zero-arg constructor for Serializer initialization if there is no constructor that accepts SparkConf.
      
      This maintains backward compatibility with older serializers implemented by users.
      f4b924f6
    • Reynold Xin's avatar
      Fall back to zero-arg constructor for Serializer initialization if there is no... · 63f90632
      Reynold Xin authored
      Fall back to zero-arg constructor for Serializer initialization if there is no constructor that accepts SparkConf.
      This maintains backward compatibility with older serializers implemented by users.
      63f90632
    • Patrick Wendell's avatar
      Provide logging when attempts to connect to the master fail. · 94fdcda8
      Patrick Wendell authored
      Without these it's a bit less clear what's going on for the user.
      
      One thing I realize when doing this is that akka itself actually retries
      the initial association. So the retry we currently have is redundant with
      akka's.
      94fdcda8
    • Patrick Wendell's avatar
      Quite akka when remote lifecycle logging is disabled. · aaaa6731
      Patrick Wendell authored
      I noticed when connecting to a standalone cluster Spark gives a bunch
      of Akka ERROR logs that make it seem like something is failing.
      
      This patch does two things:
      
      1. Akka dead letter logging is turned on/off according to the existing
         lifecycle spark property.
      2. We explicitly silence akka's EndpointWriter log in log4j. This is necessary
         because for some reason that log doesn't pick up on the lifecycle
         logging settings. After a few hours of debugging this was the only solution
         I found that worked.
      aaaa6731
    • Patrick Wendell's avatar
      Removing SPARK_EXAMPLES_JAR in the code · 79f52809
      Patrick Wendell authored
      79f52809
  4. Jan 04, 2014
  5. Jan 03, 2014
  6. Jan 02, 2014
Loading