-
- Downloads
[SPARK-10195] [SQL] Data sources Filter should not expose internal types
Spark SQL's data sources API exposes Catalyst's internal types through its Filter interfaces. This is a problem because types like UTF8String are not stable developer APIs and should not be exposed to third-parties. This issue caused incompatibilities when upgrading our `spark-redshift` library to work against Spark 1.5.0. To avoid these issues in the future we should only expose public types through these Filter objects. This patch accomplishes this by using CatalystTypeConverters to add the appropriate conversions. Author: Josh Rosen <joshrosen@databricks.com> Closes #8403 from JoshRosen/datasources-internal-vs-external-types.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala 35 additions, 32 deletions.../spark/sql/execution/datasources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala 1 addition, 1 deletion...apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala 11 additions, 8 deletions...rk/sql/execution/datasources/parquet/ParquetFilters.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/FilteredScanSuite.scala 7 additions, 0 deletions...cala/org/apache/spark/sql/sources/FilteredScanSuite.scala
Loading
Please register or sign in to comment