-
- Downloads
[SPARK-2514] [mllib] Random RDD generator
Utilities for generating random RDDs. RandomRDD and RandomVectorRDD are created instead of using `sc.parallelize(range:Range)` because `Range` objects in Scala can only have `size <= Int.MaxValue`. The object `RandomRDDGenerators` can be transformed into a generator class to reduce the number of auxiliary methods for optional arguments. Author: Doris Xin <doris.s.xin@gmail.com> Closes #1520 from dorx/randomRDD and squashes the following commits: 01121ac [Doris Xin] reviewer comments 6bf27d8 [Doris Xin] Merge branch 'master' into randomRDD a8ea92d [Doris Xin] Reviewer comments 063ea0b [Doris Xin] Merge branch 'master' into randomRDD aec68eb [Doris Xin] newline bc90234 [Doris Xin] units passed. d56cacb [Doris Xin] impl with RandomRDD 92d6f1c [Doris Xin] solution for Cloneable df5bcff [Doris Xin] Merge branch 'generator' into randomRDD f46d928 [Doris Xin] WIP 49ed20d [Doris Xin] alternative poisson distribution generator 7cb0e40 [Doris Xin] fix for data inconsistency 8881444 [Doris Xin] RandomRDDGenerator: initial design
Showing
- mllib/src/main/scala/org/apache/spark/mllib/random/DistributionGenerator.scala 101 additions, 0 deletions...org/apache/spark/mllib/random/DistributionGenerator.scala
- mllib/src/main/scala/org/apache/spark/mllib/random/RandomRDDGenerators.scala 473 additions, 0 deletions...a/org/apache/spark/mllib/random/RandomRDDGenerators.scala
- mllib/src/main/scala/org/apache/spark/mllib/rdd/RandomRDD.scala 118 additions, 0 deletions...src/main/scala/org/apache/spark/mllib/rdd/RandomRDD.scala
- mllib/src/test/scala/org/apache/spark/mllib/random/DistributionGeneratorSuite.scala 90 additions, 0 deletions...pache/spark/mllib/random/DistributionGeneratorSuite.scala
- mllib/src/test/scala/org/apache/spark/mllib/random/RandomRDDGeneratorsSuite.scala 158 additions, 0 deletions.../apache/spark/mllib/random/RandomRDDGeneratorsSuite.scala
Loading
Please register or sign in to comment