-
- Downloads
Merge pull request #408 from pwendell/external-serializers
Improvements to external sorting 1. Adds the option of compressing outputs. 2. Adds batching to the serialization to prevent OOM on the read side. 3. Slight renaming of config options. 4. Use Spark's buffer size for reads in addition to writes.
No related branches found
No related tags found
Showing
- core/src/main/scala/org/apache/spark/Aggregator.scala 1 addition, 1 deletioncore/src/main/scala/org/apache/spark/Aggregator.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 3 additions, 0 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala 51 additions, 10 deletions.../apache/spark/util/collection/ExternalAppendOnlyMap.scala
- docs/configuration.md 9 additions, 2 deletionsdocs/configuration.md
Loading
Please register or sign in to comment