Skip to content
Snippets Groups Projects
Commit 8dc44bfa authored by Mosharaf Chowdhury's avatar Mosharaf Chowdhury
Browse files

CustomBlockedInMemoryShuffle is an in- memroy implementation of CustomBlockedLFS

parent a0648358
No related branches found
No related tags found
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
conf/java-opts
+ 1
− 1
View file @ 8dc44bfa
-Dspark.shuffle.class=spark.CustomBlockedLocalFileShuffle -Dspark.blockedLocalFileShuffle.maxRxConnections=2 -Dspark.blockedLocalFileShuffle.maxTxConnections=2 -Dspark.blockedLocalFileShuffle.blockSize=256 -Dspark.blockedLocalFileShuffle.minKnockInterval=50 -Dspark.parallelLocalFileShuffle.maxRxConnections=2 -Dspark.parallelLocalFileShuffle.maxTxConnections=2 -Dspark.parallelLocalFileShuffle.minKnockInterval=50 -Dspark.parallelLocalFileShuffle.maxKnockInterval=2000 -Dspark.parallelInMemoryShuffle.maxRxConnections=2 -Dspark.parallelInMemoryShuffle.maxTxConnections=2 -Dspark.parallelInMemoryShuffle.minKnockInterval=50 -Dspark.parallelInMemoryShuffle.maxKnockInterval=2000 -Dspark.shuffle.class=spark.CustomBlockedInMemoryShuffle -Dspark.blockedLocalFileShuffle.maxRxConnections=2 -Dspark.blockedLocalFileShuffle.maxTxConnections=2 -Dspark.blockedLocalFileShuffle.blockSize=256 -Dspark.blockedLocalFileShuffle.minKnockInterval=50 -Dspark.blockedInMemoryShuffle.maxRxConnections=2 -Dspark.blockedInMemoryShuffle.maxTxConnections=2 -Dspark.blockedInMemoryShuffle.minKnockInterval=50 -Dspark.blockedInMemoryShuffle.maxKnockInterval=2000 -Dspark.parallelLocalFileShuffle.maxRxConnections=2 -Dspark.parallelLocalFileShuffle.maxTxConnections=2 -Dspark.parallelLocalFileShuffle.minKnockInterval=50 -Dspark.parallelLocalFileShuffle.maxKnockInterval=2000 -Dspark.parallelInMemoryShuffle.maxRxConnections=2 -Dspark.parallelInMemoryShuffle.maxTxConnections=2 -Dspark.parallelInMemoryShuffle.minKnockInterval=50 -Dspark.parallelInMemoryShuffle.maxKnockInterval=2000
src/scala/spark/CustomBlockedInMemoryShuffle.scala 0 → 100644
+ 657
− 0
View file @ 8dc44bfa
This diff is collapsed.
src/scala/spark/CustomParallelInMemoryShuffle.scala
+ 1
− 1
View file @ 8dc44bfa
...@@ -506,7 +506,7 @@ object CustomParallelInMemoryShuffle extends Logging { ...@@ -506,7 +506,7 @@ object CustomParallelInMemoryShuffle extends Logging {
shuffleDir + "/" + ois.readObject.asInstanceOf[String] shuffleDir + "/" + ois.readObject.asInstanceOf[String]
logInfo("requestedSplit: " + requestedSplit) logInfo("requestedSplit: " + requestedSplit)
// Send the lendth of the requestedSplit to let the receiver know that // Send the length of the requestedSplit to let the receiver know that
// transfer is about to start // transfer is about to start
// In the case of receiver timeout and connection close, this will // In the case of receiver timeout and connection close, this will
// throw a java.net.SocketException: Broken pipe // throw a java.net.SocketException: Broken pipe
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment