-
- Downloads
[SPARK-11217][ML] save/load for non-meta estimators and transformers
This PR implements the default save/load for non-meta estimators and transformers using the JSON serialization of param values. The saved metadata includes: * class name * uid * timestamp * paramMap The save/load interface is similar to DataFrames. We use the current active context by default, which should be sufficient for most use cases. ~~~scala instance.save("path") instance.write.context(sqlContext).overwrite().save("path") Instance.load("path") ~~~ The param handling is different from the design doc. We didn't save default and user-set params separately, and when we load it back, all parameters are user-set. This does cause issues. But it also cause other issues if we modify the default params. TODOs: * [x] Java test * [ ] a follow-up PR to implement default save/load for all non-meta estimators and transformers cc jkbradley Author: Xiangrui Meng <meng@databricks.com> Closes #9454 from mengxr/SPARK-11217.
Showing
- mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala 9 additions, 2 deletions...rc/main/scala/org/apache/spark/ml/feature/Binarizer.scala
- mllib/src/main/scala/org/apache/spark/ml/param/params.scala 1 addition, 1 deletionmllib/src/main/scala/org/apache/spark/ml/param/params.scala
- mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala 220 additions, 0 deletions...b/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala
- mllib/src/test/java/org/apache/spark/ml/util/JavaDefaultReadWriteSuite.java 74 additions, 0 deletions...a/org/apache/spark/ml/util/JavaDefaultReadWriteSuite.java
- mllib/src/test/scala/org/apache/spark/ml/feature/BinarizerSuite.scala 10 additions, 1 deletion...st/scala/org/apache/spark/ml/feature/BinarizerSuite.scala
- mllib/src/test/scala/org/apache/spark/ml/util/DefaultReadWriteTest.scala 110 additions, 0 deletions...scala/org/apache/spark/ml/util/DefaultReadWriteTest.scala
- mllib/src/test/scala/org/apache/spark/ml/util/TempDirectory.scala 45 additions, 0 deletions...c/test/scala/org/apache/spark/ml/util/TempDirectory.scala
Loading
Please register or sign in to comment