-
- Downloads
[SPARK-9381] [SQL] Migrate JSON data source to the new partitioning data source
Support partitioning for the JSON data source. Still 2 open issues for the `HadoopFsRelation` - `refresh()` will invoke the `discoveryPartition()`, which will auto infer the data type for the partition columns, and maybe conflict with the given partition columns. (TODO enable `HadoopFsRelationSuite.Partition column type casting" - When insert data into a cached HadoopFsRelation based table, we need to invalidate the cache after the insertion (TODO enable `InsertSuite.Caching`) Author: Cheng Hao <hao.cheng@intel.com> Closes #7696 from chenghao-intel/json and squashes the following commits: d90b104 [Cheng Hao] revert the change for JacksonGenerator.apply 307111d [Cheng Hao] fix bug in the unit test 8738c8a [Cheng Hao] fix bug in unit testing 35f2cde [Cheng Hao] support partition for json format
Showing
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 0 additions, 2 deletionssql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala 1 addition, 1 deletion...src/main/scala/org/apache/spark/sql/DataFrameReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala 2 additions, 0 deletions.../spark/sql/execution/datasources/DataSourceStrategy.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/commands.scala 8 additions, 2 deletions...org/apache/spark/sql/execution/datasources/commands.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala 13 additions, 1 deletion...la/org/apache/spark/sql/execution/datasources/rules.scala
- sql/core/src/main/scala/org/apache/spark/sql/json/JSONRelation.scala 124 additions, 161 deletions...c/main/scala/org/apache/spark/sql/json/JSONRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/json/JacksonGenerator.scala 58 additions, 0 deletions...in/scala/org/apache/spark/sql/json/JacksonGenerator.scala
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 5 additions, 4 deletions.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala 82 additions, 14 deletions.../src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/json/TestJsonData.scala 4 additions, 0 deletions...c/test/scala/org/apache/spark/sql/json/TestJsonData.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala 32 additions, 15 deletions...test/scala/org/apache/spark/sql/sources/InsertSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala 14 additions, 13 deletions...st/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala 6 additions, 9 deletions...org/apache/spark/sql/hive/MetastoreDataSourcesSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/SimpleTextHadoopFsRelationSuite.scala 30 additions, 0 deletions...e/spark/sql/sources/SimpleTextHadoopFsRelationSuite.scala
- sql/hive/src/test/scala/org/apache/spark/sql/sources/hadoopFsRelationSuites.scala 3 additions, 1 deletion...org/apache/spark/sql/sources/hadoopFsRelationSuites.scala
Loading
Please register or sign in to comment