-
- Downloads
[SPARK-11564][SQL] Dataset Java API audit
A few changes: 1. Removed fold, since it can be confusing for distributed collections. 2. Created specific interfaces for each Dataset function (e.g. MapFunction, ReduceFunction, MapPartitionsFunction) 3. Added more documentation and test cases. The other thing I'm considering doing is to have a "collector" interface for FlatMapFunction and MapPartitionsFunction, similar to MapReduce's map function. Author: Reynold Xin <rxin@databricks.com> Closes #9531 from rxin/SPARK-11564.
Showing
- core/src/main/java/org/apache/spark/api/java/function/FilterFunction.java 29 additions, 0 deletions...va/org/apache/spark/api/java/function/FilterFunction.java
- core/src/main/java/org/apache/spark/api/java/function/ForeachFunction.java 29 additions, 0 deletions...a/org/apache/spark/api/java/function/ForeachFunction.java
- core/src/main/java/org/apache/spark/api/java/function/ForeachPartitionFunction.java 28 additions, 0 deletions...che/spark/api/java/function/ForeachPartitionFunction.java
- core/src/main/java/org/apache/spark/api/java/function/Function0.java 1 addition, 1 deletion...in/java/org/apache/spark/api/java/function/Function0.java
- core/src/main/java/org/apache/spark/api/java/function/MapFunction.java 27 additions, 0 deletions.../java/org/apache/spark/api/java/function/MapFunction.java
- core/src/main/java/org/apache/spark/api/java/function/MapPartitionsFunction.java 28 additions, 0 deletions...apache/spark/api/java/function/MapPartitionsFunction.java
- core/src/main/java/org/apache/spark/api/java/function/ReduceFunction.java 27 additions, 0 deletions...va/org/apache/spark/api/java/function/ReduceFunction.java
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/Encoder.scala 32 additions, 6 deletions...cala/org/apache/spark/sql/catalyst/encoders/Encoder.scala
- sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala 36 additions, 11 deletionssql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala
- sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 49 additions, 51 deletionssql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
- sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java 7 additions, 0 deletions...st/java/test/org/apache/spark/sql/JavaDataFrameSuite.java
- sql/core/src/test/java/test/org/apache/spark/sql/JavaDatasetSuite.java 18 additions, 18 deletions...test/java/test/org/apache/spark/sql/JavaDatasetSuite.java
- sql/core/src/test/scala/org/apache/spark/sql/DatasetPrimitiveSuite.scala 0 additions, 5 deletions...st/scala/org/apache/spark/sql/DatasetPrimitiveSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala 5 additions, 5 deletions...re/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
Loading
Please register or sign in to comment