-
- Downloads
[SPARK-11569][ML] Fix StringIndexer to handle null value properly
## What changes were proposed in this pull request? This PR is to enhance StringIndexer with NULL values handling. Before the PR, StringIndexer will throw an exception when encounters NULL values. With this PR: - handleInvalid=error: Throw an exception as before - handleInvalid=skip: Skip null values as well as unseen labels - handleInvalid=keep: Give null values an additional index as well as unseen labels BTW, I noticed someone was trying to solve the same problem ( #9920 ) but seems getting no progress or response for a long time. Would you mind to give me a chance to solve it ? I'm eager to help. :-) ## How was this patch tested? new unit tests Author: Menglong TAN <tanmenglong@renrenche.com> Author: Menglong TAN <tanmenglong@gmail.com> Closes #17233 from crackcell/11569_StringIndexer_NULL.
Showing
- mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala 32 additions, 22 deletions...ain/scala/org/apache/spark/ml/feature/StringIndexer.scala
- mllib/src/test/scala/org/apache/spark/ml/feature/StringIndexerSuite.scala 45 additions, 0 deletions...cala/org/apache/spark/ml/feature/StringIndexerSuite.scala
Loading
Please register or sign in to comment