From 0077bfcb93832d93009f73f4b80f2e3d98fd2fa4 Mon Sep 17 00:00:00 2001
From: Dongjoon Hyun <dongjoon@apache.org>
Date: Wed, 8 Feb 2017 12:21:49 +0000
Subject: [PATCH] [SPARK-19409][BUILD][TEST-MAVEN] Fix
ParquetAvroCompatibilitySuite failure due to test dependency on avro
## What changes were proposed in this pull request?
After using Apache Parquet 1.8.2, `ParquetAvroCompatibilitySuite` fails on **Maven** test. It is because `org.apache.parquet.avro.AvroParquetWriter` in the test code used new `avro 1.8.0` specific class, `LogicalType`. This PR aims to fix the test dependency of `sql/core` module to use avro 1.8.0.
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-2.7/2530/consoleFull
```
ParquetAvroCompatibilitySuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: org/apache/avro/LogicalType
at org.apache.parquet.avro.AvroParquetWriter.writeSupport(AvroParquetWriter.java:144)
```
## How was this patch tested?
Pass the existing test with **Maven**.
```
$ build/mvn -Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive -Phive-thriftserver test
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:07 h
[INFO] Finished at: 2017-02-04T05:41:43+00:00
[INFO] Final Memory: 77M/987M
[INFO] ------------------------------------------------------------------------
```
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes #16795 from dongjoon-hyun/SPARK-19409-2.
---
resource-managers/mesos/pom.xml | 7 +++++++
sql/core/pom.xml | 13 +++++++++++++
2 files changed, 20 insertions(+)
diff --git a/resource-managers/mesos/pom.xml b/resource-managers/mesos/pom.xml
index c0a8f9a344..03846d9f5a 100644
--- a/resource-managers/mesos/pom.xml
+++ b/resource-managers/mesos/pom.xml
@@ -48,6 +48,13 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
<dependency>
<groupId>org.apache.mesos</groupId>
<artifactId>mesos</artifactId>
diff --git a/sql/core/pom.xml b/sql/core/pom.xml
index b8aa698090..69d797b479 100644
--- a/sql/core/pom.xml
+++ b/sql/core/pom.xml
@@ -129,6 +129,19 @@
<artifactId>parquet-avro</artifactId>
<scope>test</scope>
</dependency>
+ <!--
+ This version of avro test-dep is different from the one defined
+ in the parent pom. The parent pom has avro 1.7.7 test-dep for Hadoop.
+ Here, ParquetAvroCompatibilitySuite uses parquet-avro's AvroParquetWriter
+ which uses avro 1.8.0+ specific API. In Maven 3, we need to have
+ this here to have different versions for the same artifact.
+ -->
+ <dependency>
+ <groupId>org.apache.avro</groupId>
+ <artifactId>avro</artifactId>
+ <version>1.8.1</version>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-core</artifactId>
--
GitLab