After the last commit the bottleneck shifted from "requesting to tracker for mappers" (now done in batches) to "notifying tracker when threads leave" (done individually)

Turned OFF timers in the reducers due to inconsistent behavior (sometimes they fire, sometimes they don't)

Implemented a tracker strategy that allows reducers to create concurrent connections proportional to their time remaining. Which connections to create is random though.

Added a tracker strategy that selects random mappers for reducers. This can be used to measure tracker overhead.

 - Implemented a new tracker strategy for shuffle where if a reducer is too fast its stalled until other catchup. Basic version is working, but more work is necessary.

CustomBlockedInMemoryShuffle is receiving multiple blocks after connecting to a mapper instead of just one.

Consumption is delayed until everything has been received.  Otherwise it interferes with network performance.

CustomBlockedLocalFileShuffle: reducers are reading multiple blocks per connections instead of just one.
Sometimes ShuffleServer fails to start for small shuffle data with small block size in local VM. No problem with large block size.

Updating reception stats before consuming. Can create trouble if there is any exception during consumption (less likely,) but this frees up splits that threads can connect to instead of idling around.

 - Fixed several bugs. (Copy-paste is the bane of coding :|)

Bug fix: tracker (running in Spark master) wasn't initializing Shuffle object and was using inconsistent config values.

Tracker framework is in place that supports pluggable tracker strategy. There are several bugs along with performance problems.

 - For larger data shuffle ShuffleServerThread gets "Broken Pipe" and ShuffleClient gets "Connection Reset"
 - There is a bug in the accounting counters of BalanceConnectionsShuffleTrackerStrategy. Some of them go below zero while decrementing which is not supposed to happen.

CustomBlockedLocalFileShuffle has been added. This is essentially ManualBlockedLocalFileShuffle with our servers.

There will be a new CustomBlockedLocalFileShuffle where 'Custom' will mean ManualBlockedLocalFileShuffle with custom server instead of jetty.

Added an in-memory implementation of CustomParalleLFS. There is a serialization/deserialization bug in the implementation.

Fixed an indexing bug in HttpBlockedLocalFileShuffle. It still doesn't work on EC2 with >5 nodes cluster.

HttpBlockedLocalFileShuffle has also been converted to have per-reducer consumption thread. Works in local mesos, but NOT on EC2 :|

Upside: No synchronized blocking on "combiners" variable. 3x faster :)
Downside: Inefficient implementation. Requiring too much temporary data. Approx. 2x increase in memory requirement :( Should be fixed at some point.

Renamed BlockedLocalFileShuffle to HttpBlockedLocalFileShuffle for merging with the mos-shuffle branch.

 - Config option related to pipelining has been removed.
 - Summary: Basic -> Pipelining / Parallel -> NO pipelining

 - Renamed some ParallelLocalFileShuffle config options for clarity.

- Changed DfsShuffle to default in RDD.scala.