Skip to content
Snippets Groups Projects
  1. Aug 26, 2016
    • Michael Gummelt's avatar
      [SPARK-16967] move mesos to module · 8e5475be
      Michael Gummelt authored
      ## What changes were proposed in this pull request?
      
      Move Mesos code into a mvn module
      
      ## How was this patch tested?
      
      unit tests
      manually submitting a client mode and cluster mode job
      spark/mesos integration test suite
      
      Author: Michael Gummelt <mgummelt@mesosphere.io>
      
      Closes #14637 from mgummelt/mesos-module.
      8e5475be
  2. Jun 09, 2016
    • Josh Rosen's avatar
      [SPARK-12712] Fix failure in ./dev/test-dependencies when run against empty .m2 cache · 921fa40b
      Josh Rosen authored
      This patch fixes a bug in `./dev/test-dependencies.sh` which caused spurious failures when the script was run on a machine with an empty `.m2` cache. The problem was that extra log output from the dependency download was conflicting with the grep / regex used to identify the classpath in the Maven output. This patch fixes this issue by adjusting the regex pattern.
      
      Tested manually with the following reproduction of the bug:
      
      ```
      rm -rf ~/.m2/repository/org/apache/commons/
      ./dev/test-dependencies.sh
      ```
      
      Author: Josh Rosen <joshrosen@databricks.com>
      
      Closes #13568 from JoshRosen/SPARK-12712.
      921fa40b
  3. Apr 27, 2016
    • Dongjoon Hyun's avatar
      [SPARK-14867][BUILD] Remove `--force` option in `build/mvn` · f405de87
      Dongjoon Hyun authored
      ## What changes were proposed in this pull request?
      
      Currently, `build/mvn` provides a convenient option, `--force`, in order to use the recommended version of maven without changing PATH environment variable. However, there were two problems.
      
      - `dev/lint-java` does not use the newly installed maven.
      
        ```bash
      $ ./build/mvn --force clean
      $ ./dev/lint-java
      Using `mvn` from path: /usr/local/bin/mvn
      ```
      - It's not easy to type `--force` option always.
      
      If '--force' option is used once, we had better prefer the installed maven recommended by Spark.
      This PR makes `build/mvn` check the existence of maven installed by `--force` option first.
      
      According to the comments, this PR aims to the followings:
      - Detect the maven version from `pom.xml`.
      - Install maven if there is no or old maven.
      - Remove `--force` option.
      
      ## How was this patch tested?
      
      Manual.
      
      ```bash
      $ ./build/mvn --force clean
      $ ./dev/lint-java
      Using `mvn` from path: /Users/dongjoon/spark/build/apache-maven-3.3.9/bin/mvn
      ...
      $ rm -rf ./build/apache-maven-3.3.9/
      $ ./dev/lint-java
      Using `mvn` from path: /usr/local/bin/mvn
      ```
      
      Author: Dongjoon Hyun <dongjoon@apache.org>
      
      Closes #12631 from dongjoon-hyun/SPARK-14867.
      f405de87
  4. Jan 15, 2016
  5. Jan 11, 2016
    • Josh Rosen's avatar
      [SPARK-12734][HOTFIX] Build changes must trigger all tests; clean after install in dep tests · a4499145
      Josh Rosen authored
      This patch fixes a build/test issue caused by the combination of #10672 and a latent issue in the original `dev/test-dependencies` script.
      
      First, changes which _only_ touched build files were not triggering full Jenkins runs, making it possible for a build change to be merged even though it could cause failures in other tests. The `root` build module now depends on `build`, so all tests will now be run whenever a build-related file is changed.
      
      I also added a `clean` step to the Maven install step in `dev/test-dependencies` in order to address an issue where the dummy JARs stuck around and caused "multiple assembly JARs found" errors in tests.
      
      /cc zsxwing
      
      Author: Josh Rosen <joshrosen@databricks.com>
      
      Closes #10704 from JoshRosen/fix-build-test-problems.
      a4499145
  6. Jan 10, 2016
    • Josh Rosen's avatar
      [SPARK-12734][BUILD] Fix Netty exclusion and use Maven Enforcer to prevent future bugs · 3ab0138b
      Josh Rosen authored
      Netty classes are published under multiple artifacts with different names, so our build needs to exclude the `io.netty:netty` and `org.jboss.netty:netty` versions of the Netty artifact. However, our existing exclusions were incomplete, leading to situations where duplicate Netty classes would wind up on the classpath and cause compile errors (or worse).
      
      This patch fixes the exclusion issue by adding more exclusions and uses Maven Enforcer's [banned dependencies](https://maven.apache.org/enforcer/enforcer-rules/bannedDependencies.html) rule to prevent these classes from accidentally being reintroduced. I also updated `dev/test-dependencies.sh` to run `mvn validate` so that the enforcer rules can run as part of pull request builds.
      
      /cc rxin srowen pwendell. I'd like to backport at least the exclusion portion of this fix to `branch-1.5` in order to fix the documentation publishing job, which fails nondeterministically due to incompatible versions of Netty classes taking precedence on the compile-time classpath.
      
      Author: Josh Rosen <rosenville@gmail.com>
      Author: Josh Rosen <joshrosen@databricks.com>
      
      Closes #10672 from JoshRosen/enforce-netty-exclusions.
      3ab0138b
  7. Jan 04, 2016
    • Josh Rosen's avatar
      [SPARK-10359][PROJECT-INFRA] Use more random number in... · 9fd7a2f0
      Josh Rosen authored
      [SPARK-10359][PROJECT-INFRA] Use more random number in dev/test-dependencies.sh; fix version switching
      
      This patch aims to fix another potential source of flakiness in the `dev/test-dependencies.sh` script.
      
      pwendell's original patch and my version used `$(date +%s | tail -c6)` to generate a suffix to use when installing temporary Spark versions into the local Maven cache, but this value only changes once per second and thus is highly collision-prone when concurrent builds launch on AMPLab Jenkins. In order to reduce the potential for conflicts, this patch updates the script to call Python's random number generator instead.
      
      I also fixed a bug in how we captured the original project version; the bug was causing the exit handler code to fail.
      
      Author: Josh Rosen <joshrosen@databricks.com>
      
      Closes #10558 from JoshRosen/build-dep-tests-round-3.
      9fd7a2f0
    • Josh Rosen's avatar
      [SPARK-12612][PROJECT-INFRA] Add missing Hadoop profiles to dev/run-tests-*.py scripts and dev/deps · 0d165ec2
      Josh Rosen authored
      There are a couple of places in the `dev/run-tests-*.py` scripts which deal with Hadoop profiles, but the set of profiles that they handle does not include all Hadoop profiles defined in our POM. Similarly, the `hadoop-2.2` and `hadoop-2.6` profiles were missing from `dev/deps`.
      
      This patch updates these scripts to include all four Hadoop profiles defined in our POM.
      
      Author: Josh Rosen <joshrosen@databricks.com>
      
      Closes #10565 from JoshRosen/add-missing-hadoop-profiles-in-test-scripts.
      0d165ec2
  8. Dec 31, 2015
  9. Dec 30, 2015
    • Josh Rosen's avatar
      [SPARK-10359] Enumerate dependencies in a file and diff against it for new pull requests · 27a42c71
      Josh Rosen authored
      This patch adds a new build check which enumerates Spark's resolved runtime classpath and saves it to a file, then diffs against that file to detect whether pull requests have introduced dependency changes. The aim of this check is to make it simpler to reason about whether pull request which modify the build have introduced new dependencies or changed transitive dependencies in a way that affects the final classpath.
      
      This supplants the checks added in SPARK-4123 / #5093, which are currently disabled due to bugs.
      
      This patch is based on pwendell's work in #8531.
      
      Closes #8531.
      
      Author: Josh Rosen <joshrosen@databricks.com>
      Author: Patrick Wendell <patrick@databricks.com>
      
      Closes #10461 from JoshRosen/SPARK-10359.
      27a42c71
Loading