-
- Downloads
[SPARK-17870][MLLIB][ML] Change statistic to pValue for SelectKBest and...
[SPARK-17870][MLLIB][ML] Change statistic to pValue for SelectKBest and SelectPercentile because of DoF difference ## What changes were proposed in this pull request? For feature selection method ChiSquareSelector, it is based on the ChiSquareTestResult.statistic (ChiSqure value) to select the features. It select the features with the largest ChiSqure value. But the Degree of Freedom (df) of ChiSqure value is different in Statistics.chiSqTest(RDD), and for different df, you cannot base on ChiSqure value to select features. So we change statistic to pValue for SelectKBest and SelectPercentile ## How was this patch tested? change existing test Author: Peng <peng.meng@intel.com> Closes #15444 from mpjlu/chisqure-bug.
Showing
- mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala 2 additions, 2 deletions.../scala/org/apache/spark/mllib/feature/ChiSqSelector.scala
- mllib/src/test/scala/org/apache/spark/ml/feature/ChiSqSelectorSuite.scala 3 additions, 3 deletions...cala/org/apache/spark/ml/feature/ChiSqSelectorSuite.scala
- mllib/src/test/scala/org/apache/spark/mllib/feature/ChiSqSelectorSuite.scala 4 additions, 4 deletions...a/org/apache/spark/mllib/feature/ChiSqSelectorSuite.scala
- python/pyspark/ml/feature.py 2 additions, 2 deletionspython/pyspark/ml/feature.py
- python/pyspark/mllib/feature.py 4 additions, 4 deletionspython/pyspark/mllib/feature.py
Loading
Please register or sign in to comment