Skip to content
Snippets Groups Projects
Commit 815e0565 authored by Mike Dusenberry's avatar Mike Dusenberry Committed by Joseph K. Bradley
Browse files

[SPARK-7985] [ML] [MLlib] [Docs] Remove "fittingParamMap" references. Updating...

[SPARK-7985] [ML] [MLlib] [Docs] Remove "fittingParamMap" references. Updating ML Doc "Estimator, Transformer, and Param" examples.

Updating ML Doc's *"Estimator, Transformer, and Param"* example to use `model.extractParamMap` instead of `model.fittingParamMap`, which no longer exists.

mengxr, I believe this addresses (part of) the *update documentation* TODO list item from [PR 5820](https://github.com/apache/spark/pull/5820

).

Author: Mike Dusenberry <dusenberrymw@gmail.com>

Closes #6514 from dusenberrymw/Fix_ML_Doc_Estimator_Transformer_Param_Example and squashes the following commits:

6366e1f [Mike Dusenberry] Updating instances of model.extractParamMap to model.parent.extractParamMap, since the Params of the parent Estimator could possibly differ from thos of the Model.
d850e0e [Mike Dusenberry] Removing all references to "fittingParamMap" throughout Spark, since it has been removed.
0480304 [Mike Dusenberry] Updating the ML Doc "Estimator, Transformer, and Param" Java example to use model.extractParamMap() instead of model.fittingParamMap(), which no longer exists.
7d34939 [Mike Dusenberry] Updating ML Doc "Estimator, Transformer, and Param" example to use model.extractParamMap instead of model.fittingParamMap, which no longer exists.

(cherry picked from commit ad06727f)
Signed-off-by: default avatarJoseph K. Bradley <joseph@databricks.com>
parent 139c8240
No related branches found
No related tags found
No related merge requests found
Showing
with 14 additions and 14 deletions
...@@ -207,7 +207,7 @@ val model1 = lr.fit(training.toDF) ...@@ -207,7 +207,7 @@ val model1 = lr.fit(training.toDF)
// we can view the parameters it used during fit(). // we can view the parameters it used during fit().
// This prints the parameter (name: value) pairs, where names are unique IDs for this // This prints the parameter (name: value) pairs, where names are unique IDs for this
// LogisticRegression instance. // LogisticRegression instance.
println("Model 1 was fit using parameters: " + model1.fittingParamMap) println("Model 1 was fit using parameters: " + model1.parent.extractParamMap)
// We may alternatively specify parameters using a ParamMap, // We may alternatively specify parameters using a ParamMap,
// which supports several methods for specifying parameters. // which supports several methods for specifying parameters.
...@@ -222,7 +222,7 @@ val paramMapCombined = paramMap ++ paramMap2 ...@@ -222,7 +222,7 @@ val paramMapCombined = paramMap ++ paramMap2
// Now learn a new model using the paramMapCombined parameters. // Now learn a new model using the paramMapCombined parameters.
// paramMapCombined overrides all parameters set earlier via lr.set* methods. // paramMapCombined overrides all parameters set earlier via lr.set* methods.
val model2 = lr.fit(training.toDF, paramMapCombined) val model2 = lr.fit(training.toDF, paramMapCombined)
println("Model 2 was fit using parameters: " + model2.fittingParamMap) println("Model 2 was fit using parameters: " + model2.parent.extractParamMap)
// Prepare test data. // Prepare test data.
val test = sc.parallelize(Seq( val test = sc.parallelize(Seq(
...@@ -289,7 +289,7 @@ LogisticRegressionModel model1 = lr.fit(training); ...@@ -289,7 +289,7 @@ LogisticRegressionModel model1 = lr.fit(training);
// we can view the parameters it used during fit(). // we can view the parameters it used during fit().
// This prints the parameter (name: value) pairs, where names are unique IDs for this // This prints the parameter (name: value) pairs, where names are unique IDs for this
// LogisticRegression instance. // LogisticRegression instance.
System.out.println("Model 1 was fit using parameters: " + model1.fittingParamMap()); System.out.println("Model 1 was fit using parameters: " + model1.parent().extractParamMap());
// We may alternatively specify parameters using a ParamMap. // We may alternatively specify parameters using a ParamMap.
ParamMap paramMap = new ParamMap(); ParamMap paramMap = new ParamMap();
...@@ -305,7 +305,7 @@ ParamMap paramMapCombined = paramMap.$plus$plus(paramMap2); ...@@ -305,7 +305,7 @@ ParamMap paramMapCombined = paramMap.$plus$plus(paramMap2);
// Now learn a new model using the paramMapCombined parameters. // Now learn a new model using the paramMapCombined parameters.
// paramMapCombined overrides all parameters set earlier via lr.set* methods. // paramMapCombined overrides all parameters set earlier via lr.set* methods.
LogisticRegressionModel model2 = lr.fit(training, paramMapCombined); LogisticRegressionModel model2 = lr.fit(training, paramMapCombined);
System.out.println("Model 2 was fit using parameters: " + model2.fittingParamMap()); System.out.println("Model 2 was fit using parameters: " + model2.parent().extractParamMap());
// Prepare test documents. // Prepare test documents.
List<LabeledPoint> localTest = Lists.newArrayList( List<LabeledPoint> localTest = Lists.newArrayList(
......
...@@ -208,7 +208,7 @@ private[ml] object GBTClassificationModel { ...@@ -208,7 +208,7 @@ private[ml] object GBTClassificationModel {
require(oldModel.algo == OldAlgo.Classification, "Cannot convert GradientBoostedTreesModel" + require(oldModel.algo == OldAlgo.Classification, "Cannot convert GradientBoostedTreesModel" +
s" with algo=${oldModel.algo} (old API) to GBTClassificationModel (new API).") s" with algo=${oldModel.algo} (old API) to GBTClassificationModel (new API).")
val newTrees = oldModel.trees.map { tree => val newTrees = oldModel.trees.map { tree =>
// parent, fittingParamMap for each tree is null since there are no good ways to set these. // parent for each tree is null since there is no good way to set this.
DecisionTreeRegressionModel.fromOld(tree, null, categoricalFeatures) DecisionTreeRegressionModel.fromOld(tree, null, categoricalFeatures)
} }
val uid = if (parent != null) parent.uid else Identifiable.randomUID("gbtc") val uid = if (parent != null) parent.uid else Identifiable.randomUID("gbtc")
......
...@@ -170,7 +170,7 @@ private[ml] object RandomForestClassificationModel { ...@@ -170,7 +170,7 @@ private[ml] object RandomForestClassificationModel {
require(oldModel.algo == OldAlgo.Classification, "Cannot convert RandomForestModel" + require(oldModel.algo == OldAlgo.Classification, "Cannot convert RandomForestModel" +
s" with algo=${oldModel.algo} (old API) to RandomForestClassificationModel (new API).") s" with algo=${oldModel.algo} (old API) to RandomForestClassificationModel (new API).")
val newTrees = oldModel.trees.map { tree => val newTrees = oldModel.trees.map { tree =>
// parent, fittingParamMap for each tree is null since there are no good ways to set these. // parent for each tree is null since there is no good way to set this.
DecisionTreeClassificationModel.fromOld(tree, null, categoricalFeatures) DecisionTreeClassificationModel.fromOld(tree, null, categoricalFeatures)
} }
val uid = if (parent != null) parent.uid else Identifiable.randomUID("rfc") val uid = if (parent != null) parent.uid else Identifiable.randomUID("rfc")
......
...@@ -198,7 +198,7 @@ private[ml] object GBTRegressionModel { ...@@ -198,7 +198,7 @@ private[ml] object GBTRegressionModel {
require(oldModel.algo == OldAlgo.Regression, "Cannot convert GradientBoostedTreesModel" + require(oldModel.algo == OldAlgo.Regression, "Cannot convert GradientBoostedTreesModel" +
s" with algo=${oldModel.algo} (old API) to GBTRegressionModel (new API).") s" with algo=${oldModel.algo} (old API) to GBTRegressionModel (new API).")
val newTrees = oldModel.trees.map { tree => val newTrees = oldModel.trees.map { tree =>
// parent, fittingParamMap for each tree is null since there are no good ways to set these. // parent for each tree is null since there is no good way to set this.
DecisionTreeRegressionModel.fromOld(tree, null, categoricalFeatures) DecisionTreeRegressionModel.fromOld(tree, null, categoricalFeatures)
} }
val uid = if (parent != null) parent.uid else Identifiable.randomUID("gbtr") val uid = if (parent != null) parent.uid else Identifiable.randomUID("gbtr")
......
...@@ -152,7 +152,7 @@ private[ml] object RandomForestRegressionModel { ...@@ -152,7 +152,7 @@ private[ml] object RandomForestRegressionModel {
require(oldModel.algo == OldAlgo.Regression, "Cannot convert RandomForestModel" + require(oldModel.algo == OldAlgo.Regression, "Cannot convert RandomForestModel" +
s" with algo=${oldModel.algo} (old API) to RandomForestRegressionModel (new API).") s" with algo=${oldModel.algo} (old API) to RandomForestRegressionModel (new API).")
val newTrees = oldModel.trees.map { tree => val newTrees = oldModel.trees.map { tree =>
// parent, fittingParamMap for each tree is null since there are no good ways to set these. // parent for each tree is null since there is no good way to set this.
DecisionTreeRegressionModel.fromOld(tree, null, categoricalFeatures) DecisionTreeRegressionModel.fromOld(tree, null, categoricalFeatures)
} }
new RandomForestRegressionModel(parent.uid, newTrees) new RandomForestRegressionModel(parent.uid, newTrees)
......
...@@ -266,7 +266,7 @@ private[ml] object DecisionTreeClassifierSuite extends FunSuite { ...@@ -266,7 +266,7 @@ private[ml] object DecisionTreeClassifierSuite extends FunSuite {
val oldTree = OldDecisionTree.train(data, oldStrategy) val oldTree = OldDecisionTree.train(data, oldStrategy)
val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses) val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses)
val newTree = dt.fit(newData) val newTree = dt.fit(newData)
// Use parent, fittingParamMap from newTree since these are not checked anyways. // Use parent from newTree since this is not checked anyways.
val oldTreeAsNew = DecisionTreeClassificationModel.fromOld( val oldTreeAsNew = DecisionTreeClassificationModel.fromOld(
oldTree, newTree.parent.asInstanceOf[DecisionTreeClassifier], categoricalFeatures) oldTree, newTree.parent.asInstanceOf[DecisionTreeClassifier], categoricalFeatures)
TreeTests.checkEqual(oldTreeAsNew, newTree) TreeTests.checkEqual(oldTreeAsNew, newTree)
......
...@@ -128,7 +128,7 @@ private object GBTClassifierSuite { ...@@ -128,7 +128,7 @@ private object GBTClassifierSuite {
val oldModel = oldGBT.run(data) val oldModel = oldGBT.run(data)
val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses = 2) val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses = 2)
val newModel = gbt.fit(newData) val newModel = gbt.fit(newData)
// Use parent, fittingParamMap from newTree since these are not checked anyways. // Use parent from newTree since this is not checked anyways.
val oldModelAsNew = GBTClassificationModel.fromOld( val oldModelAsNew = GBTClassificationModel.fromOld(
oldModel, newModel.parent.asInstanceOf[GBTClassifier], categoricalFeatures) oldModel, newModel.parent.asInstanceOf[GBTClassifier], categoricalFeatures)
TreeTests.checkEqual(oldModelAsNew, newModel) TreeTests.checkEqual(oldModelAsNew, newModel)
......
...@@ -158,7 +158,7 @@ private object RandomForestClassifierSuite { ...@@ -158,7 +158,7 @@ private object RandomForestClassifierSuite {
data, oldStrategy, rf.getNumTrees, rf.getFeatureSubsetStrategy, rf.getSeed.toInt) data, oldStrategy, rf.getNumTrees, rf.getFeatureSubsetStrategy, rf.getSeed.toInt)
val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses) val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses)
val newModel = rf.fit(newData) val newModel = rf.fit(newData)
// Use parent, fittingParamMap from newTree since these are not checked anyways. // Use parent from newTree since this is not checked anyways.
val oldModelAsNew = RandomForestClassificationModel.fromOld( val oldModelAsNew = RandomForestClassificationModel.fromOld(
oldModel, newModel.parent.asInstanceOf[RandomForestClassifier], categoricalFeatures) oldModel, newModel.parent.asInstanceOf[RandomForestClassifier], categoricalFeatures)
TreeTests.checkEqual(oldModelAsNew, newModel) TreeTests.checkEqual(oldModelAsNew, newModel)
......
...@@ -83,7 +83,7 @@ private[ml] object DecisionTreeRegressorSuite extends FunSuite { ...@@ -83,7 +83,7 @@ private[ml] object DecisionTreeRegressorSuite extends FunSuite {
val oldTree = OldDecisionTree.train(data, oldStrategy) val oldTree = OldDecisionTree.train(data, oldStrategy)
val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses = 0) val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses = 0)
val newTree = dt.fit(newData) val newTree = dt.fit(newData)
// Use parent, fittingParamMap from newTree since these are not checked anyways. // Use parent from newTree since this is not checked anyways.
val oldTreeAsNew = DecisionTreeRegressionModel.fromOld( val oldTreeAsNew = DecisionTreeRegressionModel.fromOld(
oldTree, newTree.parent.asInstanceOf[DecisionTreeRegressor], categoricalFeatures) oldTree, newTree.parent.asInstanceOf[DecisionTreeRegressor], categoricalFeatures)
TreeTests.checkEqual(oldTreeAsNew, newTree) TreeTests.checkEqual(oldTreeAsNew, newTree)
......
...@@ -129,7 +129,7 @@ private object GBTRegressorSuite { ...@@ -129,7 +129,7 @@ private object GBTRegressorSuite {
val oldModel = oldGBT.run(data) val oldModel = oldGBT.run(data)
val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses = 0) val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses = 0)
val newModel = gbt.fit(newData) val newModel = gbt.fit(newData)
// Use parent, fittingParamMap from newTree since these are not checked anyways. // Use parent from newTree since this is not checked anyways.
val oldModelAsNew = GBTRegressionModel.fromOld( val oldModelAsNew = GBTRegressionModel.fromOld(
oldModel, newModel.parent.asInstanceOf[GBTRegressor], categoricalFeatures) oldModel, newModel.parent.asInstanceOf[GBTRegressor], categoricalFeatures)
TreeTests.checkEqual(oldModelAsNew, newModel) TreeTests.checkEqual(oldModelAsNew, newModel)
......
...@@ -114,7 +114,7 @@ private object RandomForestRegressorSuite extends FunSuite { ...@@ -114,7 +114,7 @@ private object RandomForestRegressorSuite extends FunSuite {
data, oldStrategy, rf.getNumTrees, rf.getFeatureSubsetStrategy, rf.getSeed.toInt) data, oldStrategy, rf.getNumTrees, rf.getFeatureSubsetStrategy, rf.getSeed.toInt)
val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses = 0) val newData: DataFrame = TreeTests.setMetadata(data, categoricalFeatures, numClasses = 0)
val newModel = rf.fit(newData) val newModel = rf.fit(newData)
// Use parent, fittingParamMap from newTree since these are not checked anyways. // Use parent from newTree since this is not checked anyways.
val oldModelAsNew = RandomForestRegressionModel.fromOld( val oldModelAsNew = RandomForestRegressionModel.fromOld(
oldModel, newModel.parent.asInstanceOf[RandomForestRegressor], categoricalFeatures) oldModel, newModel.parent.asInstanceOf[RandomForestRegressor], categoricalFeatures)
TreeTests.checkEqual(oldModelAsNew, newModel) TreeTests.checkEqual(oldModelAsNew, newModel)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment