-
- Downloads
Merge pull request #57 from aarondav/bid
Refactor BlockId into an actual type Converts all of our BlockId strings into actual BlockId types. Here are some advantages of doing this now: + Type safety + Code clarity - it's now obvious what the key of a shuffle or rdd block is, for instance. Additionally, appearing in tuple/map type signatures is a big readability bonus. A Seq[(String, BlockStatus)] is not very clear. Further, we can now use more Scala features, like matching on BlockId types. + Explicit usage - we can now formally tell where various BlockIds are being used (without doing string searches); this makes updating current BlockIds a much clearer process, and compiler-supported. (I'm looking at you, shuffle file consolidation.) + It will only get harder to make this change as time goes on. Downside is, of course, that this is a very invasive change touching a lot of different files, which will inevitably lead to merge conflicts for many.
No related branches found
No related tags found
Showing
- core/src/main/java/org/apache/spark/network/netty/FileClientHandler.java 2 additions, 1 deletion...ava/org/apache/spark/network/netty/FileClientHandler.java
- core/src/main/java/org/apache/spark/network/netty/FileServerHandler.java 4 additions, 2 deletions...ava/org/apache/spark/network/netty/FileServerHandler.java
- core/src/main/scala/org/apache/spark/BlockStoreShuffleFetcher.scala 5 additions, 6 deletions...ain/scala/org/apache/spark/BlockStoreShuffleFetcher.scala
- core/src/main/scala/org/apache/spark/CacheManager.scala 4 additions, 4 deletionscore/src/main/scala/org/apache/spark/CacheManager.scala
- core/src/main/scala/org/apache/spark/broadcast/BitTorrentBroadcast.scala 2 additions, 2 deletions...cala/org/apache/spark/broadcast/BitTorrentBroadcast.scala
- core/src/main/scala/org/apache/spark/broadcast/HttpBroadcast.scala 5 additions, 6 deletions...main/scala/org/apache/spark/broadcast/HttpBroadcast.scala
- core/src/main/scala/org/apache/spark/broadcast/TreeBroadcast.scala 3 additions, 5 deletions...main/scala/org/apache/spark/broadcast/TreeBroadcast.scala
- core/src/main/scala/org/apache/spark/executor/Executor.scala 2 additions, 2 deletionscore/src/main/scala/org/apache/spark/executor/Executor.scala
- core/src/main/scala/org/apache/spark/network/netty/FileHeader.scala 10 additions, 12 deletions...ain/scala/org/apache/spark/network/netty/FileHeader.scala
- core/src/main/scala/org/apache/spark/network/netty/ShuffleCopier.scala 14 additions, 13 deletions.../scala/org/apache/spark/network/netty/ShuffleCopier.scala
- core/src/main/scala/org/apache/spark/network/netty/ShuffleSender.scala 5 additions, 4 deletions.../scala/org/apache/spark/network/netty/ShuffleSender.scala
- core/src/main/scala/org/apache/spark/rdd/BlockRDD.scala 3 additions, 3 deletionscore/src/main/scala/org/apache/spark/rdd/BlockRDD.scala
- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala 3 additions, 3 deletions.../main/scala/org/apache/spark/scheduler/DAGScheduler.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala 2 additions, 1 deletion...rc/main/scala/org/apache/spark/scheduler/TaskResult.scala
- core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala 5 additions, 5 deletions...in/scala/org/apache/spark/serializer/KryoSerializer.scala
- core/src/main/scala/org/apache/spark/storage/BlockException.scala 1 addition, 1 deletion.../main/scala/org/apache/spark/storage/BlockException.scala
- core/src/main/scala/org/apache/spark/storage/BlockFetcherIterator.scala 12 additions, 12 deletions...scala/org/apache/spark/storage/BlockFetcherIterator.scala
- core/src/main/scala/org/apache/spark/storage/BlockId.scala 96 additions, 0 deletionscore/src/main/scala/org/apache/spark/storage/BlockId.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 42 additions, 53 deletions...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala 4 additions, 4 deletions...n/scala/org/apache/spark/storage/BlockManagerMaster.scala
Loading
Please register or sign in to comment