Skip to content
Snippets Groups Projects
Commit 230bbeaa authored by JeremyNixon's avatar JeremyNixon Committed by Xiangrui Meng
Browse files

[SPARK-10759][ML] update cross validator with include_example

This pull request uses {%include_example%} to add an example for the python cross validator to ml-guide.

Author: JeremyNixon <jnixon2@gmail.com>

Closes #11240 from JeremyNixon/pipeline_include_example.
parent 8d29001d
No related branches found
No related tags found
No related merge requests found
...@@ -283,6 +283,11 @@ However, it is also a well-established method for choosing parameters which is m ...@@ -283,6 +283,11 @@ However, it is also a well-established method for choosing parameters which is m
{% include_example java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java %} {% include_example java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java %}
</div> </div>
<div data-lang="python">
{% include_example python/ml/cross_validator.py %}
</div>
</div> </div>
## Example: model selection via train validation split ## Example: model selection via train validation split
......
...@@ -18,12 +18,14 @@ ...@@ -18,12 +18,14 @@
from __future__ import print_function from __future__ import print_function
from pyspark import SparkContext from pyspark import SparkContext
# $example on$
from pyspark.ml import Pipeline from pyspark.ml import Pipeline
from pyspark.ml.classification import LogisticRegression from pyspark.ml.classification import LogisticRegression
from pyspark.ml.evaluation import BinaryClassificationEvaluator from pyspark.ml.evaluation import BinaryClassificationEvaluator
from pyspark.ml.feature import HashingTF, Tokenizer from pyspark.ml.feature import HashingTF, Tokenizer
from pyspark.ml.tuning import CrossValidator, ParamGridBuilder from pyspark.ml.tuning import CrossValidator, ParamGridBuilder
from pyspark.sql import Row, SQLContext from pyspark.sql import Row, SQLContext
# $example off$
""" """
A simple example demonstrating model selection using CrossValidator. A simple example demonstrating model selection using CrossValidator.
...@@ -36,7 +38,7 @@ Run with: ...@@ -36,7 +38,7 @@ Run with:
if __name__ == "__main__": if __name__ == "__main__":
sc = SparkContext(appName="CrossValidatorExample") sc = SparkContext(appName="CrossValidatorExample")
sqlContext = SQLContext(sc) sqlContext = SQLContext(sc)
# $example on$
# Prepare training documents, which are labeled. # Prepare training documents, which are labeled.
LabeledDocument = Row("id", "text", "label") LabeledDocument = Row("id", "text", "label")
training = sc.parallelize([(0, "a b c d e spark", 1.0), training = sc.parallelize([(0, "a b c d e spark", 1.0),
...@@ -92,5 +94,6 @@ if __name__ == "__main__": ...@@ -92,5 +94,6 @@ if __name__ == "__main__":
selected = prediction.select("id", "text", "probability", "prediction") selected = prediction.select("id", "text", "probability", "prediction")
for row in selected.collect(): for row in selected.collect():
print(row) print(row)
# $example off$
sc.stop() sc.stop()
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment