Skip to content
Snippets Groups Projects
Commit a76bde9d authored by Holden Karau's avatar Holden Karau Committed by Andrew Or
Browse files

[SPARK-10469] [DOC] Try and document the three options

From JIRA:
Add documentation for tungsten-sort.
From the mailing list "I saw a new "spark.shuffle.manager=tungsten-sort" implemented in
https://issues.apache.org/jira/browse/SPARK-7081, but it can't be found its
corresponding description in
http://people.apache.org/~pwendell/spark-releases/spark-1.5.0-rc3-docs/configuration.html(Currenlty
there are only 'sort' and 'hash' two options)."

Author: Holden Karau <holden@pigscanfly.ca>

Closes #8638 from holdenk/SPARK-10469-document-tungsten-sort.
parent e0481113
No related branches found
No related tags found
No related merge requests found
......@@ -447,9 +447,12 @@ Apart from these, the following properties are also available, and may be useful
<td><code>spark.shuffle.manager</code></td>
<td>sort</td>
<td>
Implementation to use for shuffling data. There are two implementations available:
<code>sort</code> and <code>hash</code>. Sort-based shuffle is more memory-efficient and is
the default option starting in 1.2.
Implementation to use for shuffling data. There are three implementations available:
<code>sort</code>, <code>hash</code> and the new (1.5+) <code>tungsten-sort</code>.
Sort-based shuffle is more memory-efficient and is the default option starting in 1.2.
Tungsten-sort is similar to the sort based shuffle, with a direct binary cache-friendly
implementation with a fall back to regular sort based shuffle if its requirements are not
met.
</td>
</tr>
<tr>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment