Skip to content
Snippets Groups Projects
Commit 055be5c6 authored by Patrick Wendell's avatar Patrick Wendell
Browse files

Merge pull request #415 from pwendell/shuffle-compress

Enable compression by default for spills
parents fdaabdc6 0984647a
No related branches found
No related tags found
No related merge requests found
...@@ -81,7 +81,7 @@ private[spark] class BlockManager( ...@@ -81,7 +81,7 @@ private[spark] class BlockManager(
// Whether to compress RDD partitions that are stored serialized // Whether to compress RDD partitions that are stored serialized
val compressRdds = conf.getBoolean("spark.rdd.compress", false) val compressRdds = conf.getBoolean("spark.rdd.compress", false)
// Whether to compress shuffle output temporarily spilled to disk // Whether to compress shuffle output temporarily spilled to disk
val compressShuffleSpill = conf.getBoolean("spark.shuffle.spill.compress", false) val compressShuffleSpill = conf.getBoolean("spark.shuffle.spill.compress", true)
val heartBeatFrequency = BlockManager.getHeartBeatFrequency(conf) val heartBeatFrequency = BlockManager.getHeartBeatFrequency(conf)
......
...@@ -156,7 +156,7 @@ Apart from these, the following properties are also available, and may be useful ...@@ -156,7 +156,7 @@ Apart from these, the following properties are also available, and may be useful
</tr> </tr>
<tr> <tr>
<td>spark.shuffle.spill.compress</td> <td>spark.shuffle.spill.compress</td>
<td>false</td> <td>true</td>
<td> <td>
Whether to compress data spilled during shuffles. Whether to compress data spilled during shuffles.
</td> </td>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment