Skip to content
Snippets Groups Projects
Commit 81e3bfc1 authored by Brian O'Neill's avatar Brian O'Neill Committed by Sean Owen
Browse files

[SPARK-14421] Upgrades protobuf dependency to 2.6.1 for the new version of KCL, and…

## What changes were proposed in this pull request?

When running with Kinesis Consumer Library (KCL), against a stream that contains aggregated data, the KCL needs access to protobuf to de-aggregate the records.   Without this patch, that results in the following error message:

```
   Caused by: java.lang.ClassNotFoundException: com.google.protobuf.ProtocolStringList
```

This PR upgrades the protobuf dependency within the kinesis-asl-assembly, and relocates that package (as not to conflict with Spark's use of 2.5.0), which fixes the above CNFE.

## How was this patch tested?

Used kinesis word count example against a stream containing aggregated data.

See: SPARK-14421

Author: Brian O'Neill <bone@alumni.brown.edu>

Closes #13054 from boneill42/protobuf-relocation-for-kcl.
parent 81bf8708
No related branches found
No related tags found
No related merge requests found
......@@ -63,7 +63,12 @@
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<scope>provided</scope>
<version>2.6.1</version>
<!--
We are being explicit about version here and overriding the
spark default of 2.5.0 because KCL appears to have introduced
a dependency on protobuf 2.6.1 somewhere between KCL 1.4.0 and 1.6.1.
-->
</dependency>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
......@@ -147,6 +152,15 @@
<include>*:*</include>
</includes>
</artifactSet>
<relocations>
<relocation>
<pattern>com.google.protobuf</pattern>
<shadedPattern>kinesis.protobuf</shadedPattern>
<includes>
<include>com.google.protobuf.**</include>
</includes>
</relocation>
</relocations>
<filters>
<filter>
<artifact>*:*</artifact>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment