-
- Downloads
[SPARK-20280][CORE] FileStatusCache Weigher integer overflow
## What changes were proposed in this pull request? Weigher.weigh needs to return Int but it is possible for an Array[FileStatus] to have size > Int.maxValue. To avoid this, the size is scaled down by a factor of 32. The maximumWeight of the cache is also scaled down by the same factor. ## How was this patch tested? New test in FileIndexSuite Author: Bogdan Raducanu <bogdan@databricks.com> Closes #17591 from bogdanrdc/SPARK-20280.
Showing
- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileStatusCache.scala 34 additions, 13 deletions...che/spark/sql/execution/datasources/FileStatusCache.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileIndexSuite.scala 16 additions, 0 deletions...ache/spark/sql/execution/datasources/FileIndexSuite.scala
Loading
Please register or sign in to comment