diff --git a/docs/ml-features.md b/docs/ml-features.md
index 3cb26443b95163ab23783c2029a2ec86e4e919f9..88fd291b4be50ab1aeadc1a792e892e91e897b63 100644
--- a/docs/ml-features.md
+++ b/docs/ml-features.md
@@ -46,14 +46,18 @@ In MLlib, we separate TF and IDF to make them flexible.
`HashingTF` is a `Transformer` which takes sets of terms and converts those sets into
fixed-length feature vectors. In text processing, a "set of terms" might be a bag of words.
`HashingTF` utilizes the [hashing trick](http://en.wikipedia.org/wiki/Feature_hashing).
-A raw feature is mapped into an index (term) by applying a hash function. Then term frequencies
+A raw feature is mapped into an index (term) by applying a hash function. The hash function
+used here is [MurmurHash 3](https://en.wikipedia.org/wiki/MurmurHash). Then term frequencies
are calculated based on the mapped indices. This approach avoids the need to compute a global
term-to-index map, which can be expensive for a large corpus, but it suffers from potential hash
collisions, where different raw features may become the same term after hashing. To reduce the
chance of collision, we can increase the target feature dimension, i.e. the number of buckets
of the hash table. Since a simple modulo is used to transform the hash function to a column index,
it is advisable to use a power of two as the feature dimension, otherwise the features will
-not be mapped evenly to the columns. The default feature dimension is `$2^{18} = 262,144$`.
+not be mapped evenly to the columns. The default feature dimension is `$2^{18} = 262,144$`.
+An optional binary toggle parameter controls term frequency counts. When set to true all nonzero
+frequency counts are set to 1. This is especially useful for discrete probabilistic models that
+model binary, rather than integer, counts.
`CountVectorizer` converts text documents to vectors of term counts. Refer to [CountVectorizer
](ml-features.html#countvectorizer) for more details.
@@ -145,9 +149,11 @@ for more details on the API.
passed to other algorithms like LDA.
During the fitting process, `CountVectorizer` will select the top `vocabSize` words ordered by
- term frequency across the corpus. An optional parameter "minDF" also affects the fitting process
+ term frequency across the corpus. An optional parameter `minDF` also affects the fitting process
by specifying the minimum number (or fraction if < 1.0) of documents a term must appear in to be
- included in the vocabulary.
+ included in the vocabulary. Another optional binary toggle parameter controls the output vector.
+ If set to true all nonzero counts are set to 1. This is especially useful for discrete probabilistic
+ models that model binary, rather than integer, counts.
**Examples**
@@ -1096,14 +1102,13 @@ for more details on the API.
## QuantileDiscretizer
`QuantileDiscretizer` takes a column with continuous features and outputs a column with binned
-categorical features.
-The bin ranges are chosen by taking a sample of the data and dividing it into roughly equal parts.
-The lower and upper bin bounds will be `-Infinity` and `+Infinity`, covering all real values.
-This attempts to find `numBuckets` partitions based on a sample of the given input data, but it may
-find fewer depending on the data sample values.
-
-Note that the result may be different every time you run it, since the sample strategy behind it is
-non-deterministic.
+categorical features. The number of bins is set by the `numBuckets` parameter.
+The bin ranges are chosen using an approximate algorithm (see the documentation for
+[approxQuantile](api/scala/index.html#org.apache.spark.sql.DataFrameStatFunctions) for a
+detailed description). The precision of the approximation can be controlled with the
+`relativeError` parameter. When set to zero, exact quantiles are calculated
+(**Note:** Computing exact quantiles is an expensive operation). The lower and upper bin bounds
+will be `-Infinity` and `+Infinity` covering all real values.
**Examples**
diff --git a/examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java b/examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java
index 16f58a852d8a24ba6b421087eaf813b158f68b2f..dd20cac6211021d2cd1d4cd3f03b925b0e07ee06 100644
--- a/examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java
+++ b/examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java
@@ -54,7 +54,12 @@ public class JavaQuantileDiscretizerExample {
});
Dataset<Row> df = spark.createDataFrame(data, schema);
-
+ // $example off$
+ // Output of QuantileDiscretizer for such small datasets can depend on the number of
+ // partitions. Here we force a single partition to ensure consistent results.
+ // Note this is not necessary for normal use cases
+ df = df.repartition(1);
+ // $example on$
QuantileDiscretizer discretizer = new QuantileDiscretizer()
.setInputCol("hour")
.setOutputCol("result")
diff --git a/examples/src/main/python/ml/quantile_discretizer_example.py b/examples/src/main/python/ml/quantile_discretizer_example.py
index 6ae7bb18f8c67a5ac44d7ee6a0ad86c16ac0ff93..5444cacd957f307cb54752321081d9d6e22b5034 100644
--- a/examples/src/main/python/ml/quantile_discretizer_example.py
+++ b/examples/src/main/python/ml/quantile_discretizer_example.py
@@ -28,11 +28,16 @@ if __name__ == "__main__":
# $example on$
data = [(0, 18.0,), (1, 19.0,), (2, 8.0,), (3, 5.0,), (4, 2.2,)]
- dataFrame = spark.createDataFrame(data, ["id", "hour"])
-
+ df = spark.createDataFrame(data, ["id", "hour"])
+ # $example off$
+ # Output of QuantileDiscretizer for such small datasets can depend on the number of
+ # partitions. Here we force a single partition to ensure consistent results.
+ # Note this is not necessary for normal use cases
+ df = df.repartition(1)
+ # $example on$
discretizer = QuantileDiscretizer(numBuckets=3, inputCol="hour", outputCol="result")
- result = discretizer.fit(dataFrame).transform(dataFrame)
+ result = discretizer.fit(df).transform(df)
result.show()
# $example off$
diff --git a/examples/src/main/scala/org/apache/spark/examples/ml/QuantileDiscretizerExample.scala b/examples/src/main/scala/org/apache/spark/examples/ml/QuantileDiscretizerExample.scala
index 1a165155941612d0e58fe2ae69a5121d3af49951..2f7e217b8fe2d7711441e18a88de2e4e9684c33f 100644
--- a/examples/src/main/scala/org/apache/spark/examples/ml/QuantileDiscretizerExample.scala
+++ b/examples/src/main/scala/org/apache/spark/examples/ml/QuantileDiscretizerExample.scala
@@ -32,8 +32,13 @@ object QuantileDiscretizerExample {
// $example on$
val data = Array((0, 18.0), (1, 19.0), (2, 8.0), (3, 5.0), (4, 2.2))
- val df = spark.createDataFrame(data).toDF("id", "hour")
-
+ var df = spark.createDataFrame(data).toDF("id", "hour")
+ // $example off$
+ // Output of QuantileDiscretizer for such small datasets can depend on the number of
+ // partitions. Here we force a single partition to ensure consistent results.
+ // Note this is not necessary for normal use cases
+ .repartition(1)
+ // $example on$
val discretizer = new QuantileDiscretizer()
.setInputCol("hour")
.setOutputCol("result")