-
- Downloads
SPARK-1057 (alternative) Remove fastutil
(This is for discussion at this point -- I'm not suggesting this should be committed.) This is what removing fastutil looks like. Much of it is straightforward, like using `java.io` buffered stream classes, and Guava for murmurhash3. Uses of the `FastByteArrayOutputStream` were a little trickier. In only one case though do I think the change to use `java.io` actually entails an extra array copy. The rest is using `OpenHashMap` and `OpenHashSet`. These are now written in terms of more scala-like operations. `OpenHashMap` is where I made three non-trivial changes to make it work, and they need review: - It is no longer private - The key must be a `ClassTag` - Unless a lot of other code changes, the key type can't enforce being a supertype of `Null` It all works and tests pass, and I think there is reason to believe it's OK from a speed perspective. But what about those last changes? Author: Sean Owen <sowen@cloudera.com> Closes #266 from srowen/SPARK-1057-alternate and squashes the following commits: 2601129 [Sean Owen] Fix Map return type error not previously caught ec65502 [Sean Owen] Updates from matei's review 00bc81e [Sean Owen] Remove use of fastutil and replace with use of java.io, spark.util and Guava classes
Showing
- core/pom.xml 0 additions, 4 deletionscore/pom.xml
- core/src/main/scala/org/apache/spark/broadcast/HttpBroadcast.scala 4 additions, 5 deletions...main/scala/org/apache/spark/broadcast/HttpBroadcast.scala
- core/src/main/scala/org/apache/spark/partial/GroupedCountEvaluator.scala 13 additions, 19 deletions...cala/org/apache/spark/partial/GroupedCountEvaluator.scala
- core/src/main/scala/org/apache/spark/rdd/RDD.scala 16 additions, 18 deletionscore/src/main/scala/org/apache/spark/rdd/RDD.scala
- core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala 2 additions, 3 deletions.../scala/org/apache/spark/scheduler/ReplayListenerBus.scala
- core/src/main/scala/org/apache/spark/scheduler/Task.scala 3 additions, 6 deletionscore/src/main/scala/org/apache/spark/scheduler/Task.scala
- core/src/main/scala/org/apache/spark/serializer/Serializer.scala 3 additions, 6 deletions...c/main/scala/org/apache/spark/serializer/Serializer.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 4 additions, 6 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala 2 additions, 4 deletions...in/scala/org/apache/spark/storage/BlockObjectWriter.scala
- core/src/main/scala/org/apache/spark/util/FileLogger.scala 2 additions, 3 deletionscore/src/main/scala/org/apache/spark/util/FileLogger.scala
- core/src/main/scala/org/apache/spark/util/SizeEstimator.scala 2 additions, 3 deletions.../src/main/scala/org/apache/spark/util/SizeEstimator.scala
- core/src/main/scala/org/apache/spark/util/collection/AppendOnlyMap.scala 3 additions, 4 deletions...cala/org/apache/spark/util/collection/AppendOnlyMap.scala
- core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala 2 additions, 3 deletions.../apache/spark/util/collection/ExternalAppendOnlyMap.scala
- core/src/main/scala/org/apache/spark/util/collection/OpenHashMap.scala 2 additions, 1 deletion.../scala/org/apache/spark/util/collection/OpenHashMap.scala
- core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala 2 additions, 2 deletions.../scala/org/apache/spark/util/collection/OpenHashSet.scala
- pom.xml 0 additions, 5 deletionspom.xml
- project/SparkBuild.scala 0 additions, 1 deletionproject/SparkBuild.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/RawTextHelper.scala 8 additions, 7 deletions...scala/org/apache/spark/streaming/util/RawTextHelper.scala
- streaming/src/main/scala/org/apache/spark/streaming/util/RawTextSender.scala 4 additions, 7 deletions...scala/org/apache/spark/streaming/util/RawTextSender.scala
Loading
Please register or sign in to comment