-
- Downloads
[SPARK-21862][ML] Add overflow check in PCA
## What changes were proposed in this pull request? add overflow check in PCA, otherwise it is possible to throw `NegativeArraySizeException` when `k` and `numFeatures` are too large. The overflow checking formula is here: https://github.com/scalanlp/breeze/blob/master/math/src/main/scala/breeze/linalg/functions/svd.scala#L87 ## How was this patch tested? N/A Author: WeichenXu <weichen.xu@databricks.com> Closes #19078 from WeichenXu123/SVD_overflow_check.
Showing
- mllib/src/main/scala/org/apache/spark/mllib/feature/PCA.scala 19 additions, 0 deletions...b/src/main/scala/org/apache/spark/mllib/feature/PCA.scala
- mllib/src/test/scala/org/apache/spark/mllib/feature/PCASuite.scala 6 additions, 0 deletions.../test/scala/org/apache/spark/mllib/feature/PCASuite.scala
Please register or sign in to comment