Skip to content
Snippets Groups Projects
Commit 15539e54 authored by Josh Rosen's avatar Josh Rosen
Browse files

[SPARK-17355] Workaround for HIVE-14684 / HiveResultSetMetaData.isSigned exception

## What changes were proposed in this pull request?

Attempting to use Spark SQL's JDBC data source against the Hive ThriftServer results in a `java.sql.SQLException: Method` not supported exception from `org.apache.hive.jdbc.HiveResultSetMetaData.isSigned`. Here are two user reports of this issue:

- https://stackoverflow.com/questions/34067686/spark-1-5-1-not-working-with-hive-jdbc-1-2-0
- https://stackoverflow.com/questions/32195946/method-not-supported-in-spark

I have filed [HIVE-14684](https://issues.apache.org/jira/browse/HIVE-14684) to attempt to fix this in Hive by implementing the isSigned method, but in the meantime / for compatibility with older JDBC drivers I think we should add special-case error handling to work around this bug.

This patch updates `JDBCRDD`'s `ResultSetMetadata` to schema conversion to catch the "Method not supported" exception from Hive and return `isSigned = true`. I believe that this is safe because, as far as I know, Hive does not support unsigned numeric types.

## How was this patch tested?

Tested manually against a Spark Thrift Server.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #14911 from JoshRosen/hive-jdbc-workaround.
parent d314677c
No related branches found
No related tags found
No related merge requests found
...@@ -136,7 +136,16 @@ object JDBCRDD extends Logging { ...@@ -136,7 +136,16 @@ object JDBCRDD extends Logging {
val typeName = rsmd.getColumnTypeName(i + 1) val typeName = rsmd.getColumnTypeName(i + 1)
val fieldSize = rsmd.getPrecision(i + 1) val fieldSize = rsmd.getPrecision(i + 1)
val fieldScale = rsmd.getScale(i + 1) val fieldScale = rsmd.getScale(i + 1)
val isSigned = rsmd.isSigned(i + 1) val isSigned = {
try {
rsmd.isSigned(i + 1)
} catch {
// Workaround for HIVE-14684:
case e: SQLException if
e.getMessage == "Method not supported" &&
rsmd.getClass.getName == "org.apache.hive.jdbc.HiveResultSetMetaData" => true
}
}
val nullable = rsmd.isNullable(i + 1) != ResultSetMetaData.columnNoNulls val nullable = rsmd.isNullable(i + 1) != ResultSetMetaData.columnNoNulls
val metadata = new MetadataBuilder() val metadata = new MetadataBuilder()
.putString("name", columnName) .putString("name", columnName)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment