-
- Downloads
[SPARK-11792] [SQL] [FOLLOW-UP] Change SizeEstimation to KnownSizeEstimation...
[SPARK-11792] [SQL] [FOLLOW-UP] Change SizeEstimation to KnownSizeEstimation and make estimatedSize return Long instead of Option[Long] https://issues.apache.org/jira/browse/SPARK-11792 The main changes include: * Renaming `SizeEstimation` to `KnownSizeEstimation`. Hopefully this new name has more information. * Making `estimatedSize` return `Long` instead of `Option[Long]`. * In `UnsaveHashedRelation`, `estimatedSize` will delegate the work to `SizeEstimator` if we have not created a `BytesToBytesMap`. Since we will put `UnsaveHashedRelation` to `BlockManager`, it is generally good to let it provide a more accurate size estimation. Also, if we do not put `BytesToBytesMap` directly into `BlockerManager`, I feel it is not really necessary to make `BytesToBytesMap` extends `KnownSizeEstimation`. Author: Yin Huai <yhuai@databricks.com> Closes #9813 from yhuai/SPARK-11792-followup.
Showing
- core/src/main/scala/org/apache/spark/util/SizeEstimator.scala 16 additions, 14 deletions.../src/main/scala/org/apache/spark/util/SizeEstimator.scala
- core/src/test/scala/org/apache/spark/util/SizeEstimatorSuite.scala 2 additions, 12 deletions...test/scala/org/apache/spark/util/SizeEstimatorSuite.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala 8 additions, 4 deletions...org/apache/spark/sql/execution/joins/HashedRelation.scala
Loading
Please register or sign in to comment