-
- Downloads
[SPARK-5097][SQL] DataFrame
This pull request redesigns the existing Spark SQL dsl, which already provides data frame like functionalities. TODOs: With the exception of Python support, other tasks can be done in separate, follow-up PRs. - [ ] Audit of the API - [ ] Documentation - [ ] More test cases to cover the new API - [x] Python support - [ ] Type alias SchemaRDD Author: Reynold Xin <rxin@databricks.com> Author: Davies Liu <davies@databricks.com> Closes #4173 from rxin/df1 and squashes the following commits: 0a1a73b [Reynold Xin] Merge branch 'df1' of github.com:rxin/spark into df1 23b4427 [Reynold Xin] Mima. 828f70d [Reynold Xin] Merge pull request #7 from davies/df 257b9e6 [Davies Liu] add repartition 6bf2b73 [Davies Liu] fix collect with UDT and tests e971078 [Reynold Xin] Missing quotes. b9306b4 [Reynold Xin] Remove removeColumn/updateColumn for now. a728bf2 [Reynold Xin] Example rename. e8aa3d3 [Reynold Xin] groupby -> groupBy. 9662c9e [Davies Liu] improve DataFrame Python API 4ae51ea [Davies Liu] python API for dataframe 1e5e454 [Reynold Xin] Fixed a bug with symbol conversion. 2ca74db [Reynold Xin] Couple minor fixes. ea98ea1 [Reynold Xin] Documentation & literal expressions. 2b22684 [Reynold Xin] Got rid of IntelliJ problems. 02bbfbc [Reynold Xin] Tightening imports. ffbce66 [Reynold Xin] Fixed compilation error. 59b6d8b [Reynold Xin] Style violation. b85edfb [Reynold Xin] ALS. 8c37f0a [Reynold Xin] Made MLlib and examples compile 6d53134 [Reynold Xin] Hive module. d35efd5 [Reynold Xin] Fixed compilation error. ce4a5d2 [Reynold Xin] Fixed test cases in SQL except ParquetIOSuite. 66d5ef1 [Reynold Xin] SQLContext minor patch. c9bcdc0 [Reynold Xin] Checkpoint: SQL module compiles!
Showing
- examples/src/main/java/org/apache/spark/examples/ml/JavaCrossValidatorExample.java 5 additions, 5 deletions...g/apache/spark/examples/ml/JavaCrossValidatorExample.java
- examples/src/main/java/org/apache/spark/examples/ml/JavaSimpleParamsExample.java 6 additions, 6 deletions...org/apache/spark/examples/ml/JavaSimpleParamsExample.java
- examples/src/main/java/org/apache/spark/examples/ml/JavaSimpleTextClassificationPipeline.java 5 additions, 5 deletions...ark/examples/ml/JavaSimpleTextClassificationPipeline.java
- examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQL.java 18 additions, 18 deletions...main/java/org/apache/spark/examples/sql/JavaSparkSQL.java
- examples/src/main/python/mllib/dataset_example.py 1 addition, 1 deletionexamples/src/main/python/mllib/dataset_example.py
- examples/src/main/python/sql.py 8 additions, 8 deletionsexamples/src/main/python/sql.py
- examples/src/main/scala/org/apache/spark/examples/ml/CrossValidatorExample.scala 1 addition, 2 deletions.../org/apache/spark/examples/ml/CrossValidatorExample.scala
- examples/src/main/scala/org/apache/spark/examples/ml/MovieLensALS.scala 1 addition, 1 deletion...ain/scala/org/apache/spark/examples/ml/MovieLensALS.scala
- examples/src/main/scala/org/apache/spark/examples/ml/SimpleParamsExample.scala 2 additions, 3 deletions...la/org/apache/spark/examples/ml/SimpleParamsExample.scala
- examples/src/main/scala/org/apache/spark/examples/ml/SimpleTextClassificationPipeline.scala 1 addition, 2 deletions.../spark/examples/ml/SimpleTextClassificationPipeline.scala
- examples/src/main/scala/org/apache/spark/examples/mllib/DatasetExample.scala 14 additions, 14 deletions...cala/org/apache/spark/examples/mllib/DatasetExample.scala
- examples/src/main/scala/org/apache/spark/examples/sql/RDDRelation.scala 4 additions, 2 deletions...ain/scala/org/apache/spark/examples/sql/RDDRelation.scala
- mllib/src/main/scala/org/apache/spark/ml/Estimator.scala 4 additions, 4 deletionsmllib/src/main/scala/org/apache/spark/ml/Estimator.scala
- mllib/src/main/scala/org/apache/spark/ml/Evaluator.scala 2 additions, 2 deletionsmllib/src/main/scala/org/apache/spark/ml/Evaluator.scala
- mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala 3 additions, 3 deletionsmllib/src/main/scala/org/apache/spark/ml/Pipeline.scala
- mllib/src/main/scala/org/apache/spark/ml/Transformer.scala 8 additions, 9 deletionsmllib/src/main/scala/org/apache/spark/ml/Transformer.scala
- mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala 6 additions, 8 deletions...g/apache/spark/ml/classification/LogisticRegression.scala
- mllib/src/main/scala/org/apache/spark/ml/evaluation/BinaryClassificationEvaluator.scala 3 additions, 4 deletions...e/spark/ml/evaluation/BinaryClassificationEvaluator.scala
- mllib/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala 5 additions, 10 deletions...in/scala/org/apache/spark/ml/feature/StandardScaler.scala
- mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala 17 additions, 20 deletions...c/main/scala/org/apache/spark/ml/recommendation/ALS.scala
Loading
Please register or sign in to comment