-
- Downloads
[SPARK-7654][SQL] DataFrameReader and DataFrameWriter for input/output API
This patch introduces DataFrameWriter and DataFrameReader. DataFrameReader interface, accessible through SQLContext.read, contains methods that create DataFrames. These methods used to reside in SQLContext. Example usage: ```scala sqlContext.read.json("...") sqlContext.read.parquet("...") ``` DataFrameWriter interface, accessible through DataFrame.write, implements a builder pattern to avoid the proliferation of options in writing DataFrame out. It currently implements: - mode - format (e.g. "parquet", "json") - options (generic options passed down into data sources) - partitionBy (partitioning columns) Example usage: ```scala df.write.mode("append").format("json").partitionBy("date").saveAsTable("myJsonTable") ``` TODO: - [ ] Documentation update - [ ] Move JDBC into reader / writer? - [ ] Deprecate the old interfaces - [ ] Move the generic load interface into reader. - [ ] Update example code and documentation Author: Reynold Xin <rxin@databricks.com> Closes #6175 from rxin/reader-writer and squashes the following commits: b146c95 [Reynold Xin] Deprecation of old APIs. bd8abdf [Reynold Xin] Fixed merge conflict. 26abea2 [Reynold Xin] Added general load methods. 244fbec [Reynold Xin] Added equivalent to example. 4f15d92 [Reynold Xin] Added documentation for partitionBy. 7e91611 [Reynold Xin] [SPARK-7654][SQL] DataFrameReader and DataFrameWriter for input/output API.
Showing
- examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQL.java 2 additions, 2 deletions...main/java/org/apache/spark/examples/sql/JavaSparkSQL.java
- examples/src/main/scala/org/apache/spark/examples/mllib/DatasetExample.scala 1 addition, 1 deletion...cala/org/apache/spark/examples/mllib/DatasetExample.scala
- examples/src/main/scala/org/apache/spark/examples/sql/RDDRelation.scala 1 addition, 1 deletion...ain/scala/org/apache/spark/examples/sql/RDDRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 38 additions, 134 deletionssql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala 218 additions, 0 deletions...src/main/scala/org/apache/spark/sql/DataFrameReader.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriter.scala 198 additions, 0 deletions...src/main/scala/org/apache/spark/sql/DataFrameWriter.scala
- sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala 52 additions, 106 deletions...core/src/main/scala/org/apache/spark/sql/SQLContext.scala
- sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTest.scala 4 additions, 4 deletions...main/scala/org/apache/spark/sql/parquet/ParquetTest.scala
- sql/core/src/test/java/test/org/apache/spark/sql/sources/JavaSaveLoadSuite.java 4 additions, 4 deletions.../test/org/apache/spark/sql/sources/JavaSaveLoadSuite.java
- sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 2 additions, 2 deletions.../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala 9 additions, 8 deletions...e/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala 2 additions, 2 deletions...est/scala/org/apache/spark/sql/UserDefinedTypeSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala 25 additions, 25 deletions.../src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetFilterSuite.scala 3 additions, 3 deletions...ala/org/apache/spark/sql/parquet/ParquetFilterSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala 20 additions, 21 deletions...t/scala/org/apache/spark/sql/parquet/ParquetIOSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/parquet/ParquetPartitionDiscoverySuite.scala 5 additions, 11 deletions...he/spark/sql/parquet/ParquetPartitionDiscoverySuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/CreateTableAsSelectSuite.scala 1 addition, 1 deletion...g/apache/spark/sql/sources/CreateTableAsSelectSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/InsertSuite.scala 5 additions, 5 deletions...test/scala/org/apache/spark/sql/sources/InsertSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala 13 additions, 13 deletions...st/scala/org/apache/spark/sql/sources/SaveLoadSuite.scala
- sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveStrategies.scala 2 additions, 2 deletions...main/scala/org/apache/spark/sql/hive/HiveStrategies.scala
Loading
Please register or sign in to comment