Skip to content
Snippets Groups Projects
  • Steve Loughran's avatar
    37d95227
    [SPARK-17058][BUILD] Add maven snapshots-and-staging profile to build/test... · 37d95227
    Steve Loughran authored
    [SPARK-17058][BUILD] Add maven snapshots-and-staging profile to build/test against staging artifacts
    
    ## What changes were proposed in this pull request?
    
    Adds a `snapshots-and-staging profile` so that  RCs of projects like Hadoop and HBase can be used in developer-only build and test runs. There's a comment above the profile telling people not to use this in production.
    
    There's no attempt to do the same for SBT, as Ivy is different.
    ## How was this patch tested?
    
    Tested by building against the Hadoop 2.7.3 RC 1 JARs
    
    without the profile (and without any local copy of the 2.7.3 artifacts), the build failed
    
    ```
    mvn install -DskipTests -Pyarn,hadoop-2.7,hive -Dhadoop.version=2.7.3
    
    ...
    
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project Launcher 2.1.0-SNAPSHOT
    [INFO] ------------------------------------------------------------------------
    Downloading: https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.7.3/hadoop-client-2.7.3.pom
    [WARNING] The POM for org.apache.hadoop:hadoop-client:jar:2.7.3 is missing, no dependency information available
    Downloading: https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.7.3/hadoop-client-2.7.3.jar
    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Summary:
    [INFO]
    [INFO] Spark Project Parent POM ........................... SUCCESS [  4.482 s]
    [INFO] Spark Project Tags ................................. SUCCESS [ 17.402 s]
    [INFO] Spark Project Sketch ............................... SUCCESS [ 11.252 s]
    [INFO] Spark Project Networking ........................... SUCCESS [ 13.458 s]
    [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  9.043 s]
    [INFO] Spark Project Unsafe ............................... SUCCESS [ 16.027 s]
    [INFO] Spark Project Launcher ............................. FAILURE [  1.653 s]
    [INFO] Spark Project Core ................................. SKIPPED
    ...
    ```
    
    With the profile, the build completed
    
    ```
    mvn install -DskipTests -Pyarn,hadoop-2.7,hive,snapshots-and-staging -Dhadoop.version=2.7.3
    ```
    
    Author: Steve Loughran <stevel@apache.org>
    
    Closes #14646 from steveloughran/stevel/SPARK-17058-support-asf-snapshots.
    37d95227
    History
    [SPARK-17058][BUILD] Add maven snapshots-and-staging profile to build/test...
    Steve Loughran authored
    [SPARK-17058][BUILD] Add maven snapshots-and-staging profile to build/test against staging artifacts
    
    ## What changes were proposed in this pull request?
    
    Adds a `snapshots-and-staging profile` so that  RCs of projects like Hadoop and HBase can be used in developer-only build and test runs. There's a comment above the profile telling people not to use this in production.
    
    There's no attempt to do the same for SBT, as Ivy is different.
    ## How was this patch tested?
    
    Tested by building against the Hadoop 2.7.3 RC 1 JARs
    
    without the profile (and without any local copy of the 2.7.3 artifacts), the build failed
    
    ```
    mvn install -DskipTests -Pyarn,hadoop-2.7,hive -Dhadoop.version=2.7.3
    
    ...
    
    [INFO] ------------------------------------------------------------------------
    [INFO] Building Spark Project Launcher 2.1.0-SNAPSHOT
    [INFO] ------------------------------------------------------------------------
    Downloading: https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.7.3/hadoop-client-2.7.3.pom
    [WARNING] The POM for org.apache.hadoop:hadoop-client:jar:2.7.3 is missing, no dependency information available
    Downloading: https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-client/2.7.3/hadoop-client-2.7.3.jar
    [INFO] ------------------------------------------------------------------------
    [INFO] Reactor Summary:
    [INFO]
    [INFO] Spark Project Parent POM ........................... SUCCESS [  4.482 s]
    [INFO] Spark Project Tags ................................. SUCCESS [ 17.402 s]
    [INFO] Spark Project Sketch ............................... SUCCESS [ 11.252 s]
    [INFO] Spark Project Networking ........................... SUCCESS [ 13.458 s]
    [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  9.043 s]
    [INFO] Spark Project Unsafe ............................... SUCCESS [ 16.027 s]
    [INFO] Spark Project Launcher ............................. FAILURE [  1.653 s]
    [INFO] Spark Project Core ................................. SKIPPED
    ...
    ```
    
    With the profile, the build completed
    
    ```
    mvn install -DskipTests -Pyarn,hadoop-2.7,hive,snapshots-and-staging -Dhadoop.version=2.7.3
    ```
    
    Author: Steve Loughran <stevel@apache.org>
    
    Closes #14646 from steveloughran/stevel/SPARK-17058-support-asf-snapshots.