Skip to content
Snippets Groups Projects
Commit 61eb1267 authored by martinzapletal's avatar martinzapletal Committed by Xiangrui Meng
Browse files

[MLLIB][SPARK-5502] User guide for isotonic regression 

User guide for isotonic regression added to docs/mllib-regression.md including code examples for Scala and Java.

Author: martinzapletal <zapletal-martin@email.cz>

Closes #4536 from zapletal-martin/SPARK-5502 and squashes the following commits:

67fe773 [martinzapletal] SPARK-5502 reworded model prediction rules to use more general language rather than the code/implementation specific terms
80bd4c3 [martinzapletal] SPARK-5502 created docs page for isotonic regression, added links to the page, updated data and examples
7d8136e [martinzapletal] SPARK-5502 Added documentation for Isotonic regression including examples for Scala and Java
504b5c3 [martinzapletal] SPARK-5502 Added documentation for Isotonic regression including examples for Scala and Java
parent c771e475
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
data/mllib/sample_isotonic_regression_data.txt 0 → 100644
+ 100
− 0
View file @ 61eb1267
0.24579296,0.01
0.28505864,0.02
0.31208567,0.03
0.35900051,0.04
0.35747068,0.05
0.16675166,0.06
0.17491076,0.07
0.04181540,0.08
0.04793473,0.09
0.03926568,0.10
0.12952575,0.11
0.00000000,0.12
0.01376849,0.13
0.13105558,0.14
0.08873024,0.15
0.12595614,0.16
0.15247323,0.17
0.25956145,0.18
0.20040796,0.19
0.19581846,0.20
0.15757267,0.21
0.13717491,0.22
0.19020908,0.23
0.19581846,0.24
0.20091790,0.25
0.16879143,0.26
0.18510964,0.27
0.20040796,0.28
0.29576747,0.29
0.43396226,0.30
0.53391127,0.31
0.52116267,0.32
0.48546660,0.33
0.49209587,0.34
0.54156043,0.35
0.59765426,0.36
0.56144824,0.37
0.58592555,0.38
0.52983172,0.39
0.50178480,0.40
0.52626211,0.41
0.58286588,0.42
0.64660887,0.43
0.68077511,0.44
0.74298827,0.45
0.64864865,0.46
0.67261601,0.47
0.65782764,0.48
0.69811321,0.49
0.63029067,0.50
0.61601224,0.51
0.63233044,0.52
0.65323814,0.53
0.65323814,0.54
0.67363590,0.55
0.67006629,0.56
0.51555329,0.57
0.50892402,0.58
0.33299337,0.59
0.36206017,0.60
0.43090260,0.61
0.45996940,0.62
0.56348802,0.63
0.54920959,0.64
0.48393677,0.65
0.48495665,0.66
0.46965834,0.67
0.45181030,0.68
0.45843957,0.69
0.47118817,0.70
0.51555329,0.71
0.58031617,0.72
0.55481897,0.73
0.56297807,0.74
0.56603774,0.75
0.57929628,0.76
0.64762876,0.77
0.66241713,0.78
0.69301377,0.79
0.65119837,0.80
0.68332483,0.81
0.66598674,0.82
0.73890872,0.83
0.73992861,0.84
0.84242733,0.85
0.91330954,0.86
0.88016318,0.87
0.90719021,0.88
0.93115757,0.89
0.93115757,0.90
0.91942886,0.91
0.92911780,0.92
0.95665477,0.93
0.95002550,0.94
0.96940337,0.95
1.00000000,0.96
0.89801122,0.97
0.90311066,0.98
0.90362060,0.99
0.83477817,1.0
\ No newline at end of file
docs/mllib-classification-regression.md
+ 2
− 1
View file @ 61eb1267
......@@ -23,7 +23,7 @@ the supported algorithms for each type of problem.
<td>Multiclass Classification</td><td>decision trees, naive Bayes</td>
</tr>
<tr>
<td>Regression</td><td>linear least squares, Lasso, ridge regression, decision trees</td>
<td>Regression</td><td>linear least squares, Lasso, ridge regression, decision trees, isotonic regression</td>
</tr>
</tbody>
</table>
......@@ -35,3 +35,4 @@ More details for these methods can be found here:
* [linear regression (least squares, Lasso, ridge)](mllib-linear-methods.html#linear-least-squares-lasso-and-ridge-regression)
* [Decision trees](mllib-decision-tree.html)
* [Naive Bayes](mllib-naive-bayes.html)
* [Isotonic regression](mllib-isotonic-regression.html)
docs/mllib-guide.md
+ 1
− 0
View file @ 61eb1267
......@@ -21,6 +21,7 @@ filtering, dimensionality reduction, as well as underlying optimization primitiv
* [naive Bayes](mllib-naive-bayes.html)
* [decision trees](mllib-decision-tree.html)
* [ensembles of trees](mllib-ensembles.html) (Random Forests and Gradient-Boosted Trees)
* [isotonic regression](mllib-isotonic-regression.html)
* [Collaborative filtering](mllib-collaborative-filtering.html)
* alternating least squares (ALS)
* [Clustering](mllib-clustering.html)
......
docs/mllib-isotonic-regression.md 0 → 100644
+ 155
− 0
View file @ 61eb1267
---
layout: global
title: Naive Bayes - MLlib
displayTitle: <a href="mllib-guide.html">MLlib</a> - Regression
---
## Isotonic regression
[Isotonic regression](http://en.wikipedia.org/wiki/Isotonic_regression)
belongs to the family of regression algorithms. Formally isotonic regression is a problem where
given a finite set of real numbers `$Y = {y_1, y_2, ..., y_n}$` representing observed responses
and `$X = {x_1, x_2, ..., x_n}$` the unknown response values to be fitted
finding a function that minimises
`\begin{equation}
f(x) = \sum_{i=1}^n w_i (y_i - x_i)^2
\end{equation}`
with respect to complete order subject to
`$x_1\le x_2\le ...\le x_n$` where `$w_i$` are positive weights.
The resulting function is called isotonic regression and it is unique.
It can be viewed as least squares problem under order restriction.
Essentially isotonic regression is a
[monotonic function](http://en.wikipedia.org/wiki/Monotonic_function)
best fitting the original data points.
MLlib supports a
[pool adjacent violators algorithm](http://doi.org/10.1198/TECH.2010.10111)
which uses an approach to
[parallelizing isotonic regression](http://doi.org/10.1007/978-3-642-99789-1_10).
The training input is a RDD of tuples of three double values that represent
label, feature and weight in this order. Additionally IsotonicRegression algorithm has one
optional parameter called $isotonic$ defaulting to true.
This argument specifies if the isotonic regression is
isotonic (monotonically increasing) or antitonic (monotonically decreasing).
Training returns an IsotonicRegressionModel that can be used to predict
labels for both known and unknown features. The result of isotonic regression
is treated as piecewise linear function. The rules for prediction therefore are:
* If the prediction input exactly matches a training feature
then associated prediction is returned. In case there are multiple predictions with the same
feature then one of them is returned. Which one is undefined
(same as java.util.Arrays.binarySearch).
* If the prediction input is lower or higher than all training features
then prediction with lowest or highest feature is returned respectively.
In case there are multiple predictions with the same feature
then the lowest or highest is returned respectively.
* If the prediction input falls between two training features then prediction is treated
as piecewise linear function and interpolated value is calculated from the
predictions of the two closest features. In case there are multiple values
with the same feature then the same rules as in previous point are used.
### Examples
<div class="codetabs">
<div data-lang="scala" markdown="1">
Data are read from a file where each line has a format label,feature
i.e. 4710.28,500.00. The data are split to training and testing set.
Model is created using the training set and a mean squared error is calculated from the predicted
labels and real labels in the test set.
{% highlight scala %}
import org.apache.spark.mllib.regression.IsotonicRegression
val data = sc.textFile("data/mllib/sample_isotonic_regression_data.txt")
// Create label, feature, weight tuples from input data with weight set to default value 1.0.
val parsedData = data.map { line =>
val parts = line.split(',').map(_.toDouble)
(parts(0), parts(1), 1.0)
}
// Split data into training (60%) and test (40%) sets.
val splits = parsedData.randomSplit(Array(0.6, 0.4), seed = 11L)
val training = splits(0)
val test = splits(1)
// Create isotonic regression model from training data.
// Isotonic parameter defaults to true so it is only shown for demonstration
val model = new IsotonicRegression().setIsotonic(true).run(training)
// Create tuples of predicted and real labels.
val predictionAndLabel = test.map { point =>
val predictedLabel = model.predict(point._2)
(predictedLabel, point._1)
}
// Calculate mean squared error between predicted and real labels.
val meanSquaredError = predictionAndLabel.map{case(p, l) => math.pow((p - l), 2)}.mean()
println("Mean Squared Error = " + meanSquaredError)
{% endhighlight %}
</div>
<div data-lang="java" markdown="1">
Data are read from a file where each line has a format label,feature
i.e. 4710.28,500.00. The data are split to training and testing set.
Model is created using the training set and a mean squared error is calculated from the predicted
labels and real labels in the test set.
{% highlight java %}
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaDoubleRDD;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.api.java.function.PairFunction;
import org.apache.spark.mllib.regression.IsotonicRegressionModel;
import scala.Tuple2;
import scala.Tuple3;
JavaRDD<String> data = sc.textFile("data/mllib/sample_isotonic_regression_data.txt");
// Create label, feature, weight tuples from input data with weight set to default value 1.0.
JavaRDD<Tuple3<Double, Double, Double>> parsedData = data.map(
new Function<String, Tuple3<Double, Double, Double>>() {
public Tuple3<Double, Double, Double> call(String line) {
String[] parts = line.split(",");
return new Tuple3<>(new Double(parts[0]), new Double(parts[1]), 1.0);
}
}
);
// Split data into training (60%) and test (40%) sets.
JavaRDD<Tuple3<Double, Double, Double>>[] splits = parsedData.randomSplit(new double[] {0.6, 0.4}, 11L);
JavaRDD<Tuple3<Double, Double, Double>> training = splits[0];
JavaRDD<Tuple3<Double, Double, Double>> test = splits[1];
// Create isotonic regression model from training data.
// Isotonic parameter defaults to true so it is only shown for demonstration
IsotonicRegressionModel model = new IsotonicRegression().setIsotonic(true).run(training);
// Create tuples of predicted and real labels.
JavaPairRDD<Double, Double> predictionAndLabel = test.mapToPair(
new PairFunction<Tuple3<Double, Double, Double>, Double, Double>() {
@Override public Tuple2<Double, Double> call(Tuple3<Double, Double, Double> point) {
Double predictedLabel = model.predict(point._2());
return new Tuple2<Double, Double>(predictedLabel, point._1());
}
}
);
// Calculate mean squared error between predicted and real labels.
Double meanSquaredError = new JavaDoubleRDD(predictionAndLabel.map(
new Function<Tuple2<Double, Double>, Object>() {
@Override public Object call(Tuple2<Double, Double> pl) {
return Math.pow(pl._1() - pl._2(), 2);
}
}
).rdd()).mean();
System.out.println("Mean Squared Error = " + meanSquaredError);
{% endhighlight %}
</div>
</div>
\ No newline at end of file
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment