-
- Downloads
[SPARK-12833][SQL] Initial import of spark-csv
CSV is the most common data format in the "small data" world. It is often the first format people want to try when they see Spark on a single node. Having to rely on a 3rd party component for this leads to poor user experience for new users. This PR merges the popular spark-csv data source package (https://github.com/databricks/spark-csv) with SparkSQL. This is a first PR to bring the functionality to spark 2.0 master. We will complete items outlines in the design document (see JIRA attachment) in follow up pull requests. Author: Hossein <hossein@databricks.com> Author: Reynold Xin <rxin@databricks.com> Closes #10766 from rxin/csv.
Showing
- .rat-excludes 2 additions, 0 deletions.rat-excludes
- NOTICE 37 additions, 1 deletionNOTICE
- dev/deps/spark-deps-hadoop-2.2 1 addition, 0 deletionsdev/deps/spark-deps-hadoop-2.2
- dev/deps/spark-deps-hadoop-2.3 1 addition, 0 deletionsdev/deps/spark-deps-hadoop-2.3
- dev/deps/spark-deps-hadoop-2.4 1 addition, 0 deletionsdev/deps/spark-deps-hadoop-2.4
- dev/deps/spark-deps-hadoop-2.6 1 addition, 0 deletionsdev/deps/spark-deps-hadoop-2.6
- sql/core/pom.xml 6 additions, 0 deletionssql/core/pom.xml
- sql/core/src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister 1 addition, 0 deletions.../services/org.apache.spark.sql.sources.DataSourceRegister
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala 227 additions, 0 deletions.../spark/sql/execution/datasources/csv/CSVInferSchema.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVParameters.scala 107 additions, 0 deletions...e/spark/sql/execution/datasources/csv/CSVParameters.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVParser.scala 243 additions, 0 deletions...pache/spark/sql/execution/datasources/csv/CSVParser.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVRelation.scala 298 additions, 0 deletions...che/spark/sql/execution/datasources/csv/CSVRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/DefaultSource.scala 48 additions, 0 deletions...e/spark/sql/execution/datasources/csv/DefaultSource.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/InferSchema.scala 6 additions, 7 deletions...he/spark/sql/execution/datasources/json/InferSchema.scala
- sql/core/src/test/resources/cars-alternative.csv 5 additions, 0 deletionssql/core/src/test/resources/cars-alternative.csv
- sql/core/src/test/resources/cars-null.csv 6 additions, 0 deletionssql/core/src/test/resources/cars-null.csv
- sql/core/src/test/resources/cars-unbalanced-quotes.csv 4 additions, 0 deletionssql/core/src/test/resources/cars-unbalanced-quotes.csv
- sql/core/src/test/resources/cars.csv 6 additions, 0 deletionssql/core/src/test/resources/cars.csv
- sql/core/src/test/resources/cars.tsv 4 additions, 0 deletionssql/core/src/test/resources/cars.tsv
- sql/core/src/test/resources/cars_iso-8859-1.csv 6 additions, 0 deletionssql/core/src/test/resources/cars_iso-8859-1.csv
Loading
Please register or sign in to comment