Skip to content
Snippets Groups Projects
Commit 4432568a authored by Yuhao Yang's avatar Yuhao Yang Committed by Xiangrui Meng
Browse files

[SPARK-5282][mllib]: RowMatrix easily gets int overflow in the memory size warning

JIRA: https://issues.apache.org/jira/browse/SPARK-5282

fix the possible int overflow in the memory computation warning

Author: Yuhao Yang <hhbyyh@gmail.com>

Closes #4069 from hhbyyh/addscStop and squashes the following commits:

e54e5c8 [Yuhao Yang] change to MB based number
7afac23 [Yuhao Yang] 5282: fix int overflow in the warning
parent 1ac1c1dc
No related branches found
No related tags found
No related merge requests found
......@@ -131,8 +131,8 @@ class RowMatrix(
throw new IllegalArgumentException(s"Argument with more than 65535 cols: $cols")
}
if (cols > 10000) {
val mem = cols * cols * 8
logWarning(s"$cols columns will require at least $mem bytes of memory!")
val memMB = (cols.toLong * cols) / 125000
logWarning(s"$cols columns will require at least $memMB megabytes of memory!")
}
}
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment