-
- Downloads
[SPARK-15740][MLLIB] Word2VecSuite "big model load / save" caused OOM in maven jenkins builds
## What changes were proposed in this pull request? "test big model load / save" in Word2VecSuite, lately resulted into OOM. Therefore we decided to make the partitioning adaptive (not based on spark default "spark.kryoserializer.buffer.max" conf) and then testing it using a small buffer size in order to trigger partitioning without allocating too much memory for the test. ## How was this patch tested? It was tested running the following unit test: org.apache.spark.mllib.feature.Word2VecSuite Author: tmnd1991 <antonio.murgia2@studio.unibo.it> Closes #13509 from tmnd1991/SPARK-15740.
Showing
- mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala 9 additions, 7 deletions.../main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
- mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala 22 additions, 3 deletions.../scala/org/apache/spark/mllib/feature/Word2VecSuite.scala
Loading
Please register or sign in to comment