Skip to content
Snippets Groups Projects
Commit 9edd8878 authored by Chen Chao's avatar Chen Chao Committed by Patrick Wendell
Browse files

update spark.default.parallelism

actually, the value 8 is only valid in mesos fine-grained mode :
<code>
  override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)
</code>

while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!
<code>
override def defaultParallelism(): Int = {
   conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
  }
</code>

Author: Chen Chao <crazyjvm@gmail.com>

Closes #389 from CrazyJvm/patch-2 and squashes the following commits:

84a7fe4 [Chen Chao] miss </li> at the end of every single line
04a9796 [Chen Chao] change format
ee0fae0 [Chen Chao] update spark.default.parallelism
parent fec462c1
No related branches found
No related tags found
No related merge requests found
...@@ -96,7 +96,13 @@ Apart from these, the following properties are also available, and may be useful ...@@ -96,7 +96,13 @@ Apart from these, the following properties are also available, and may be useful
<tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr> <tr><th>Property Name</th><th>Default</th><th>Meaning</th></tr>
<tr> <tr>
<td>spark.default.parallelism</td> <td>spark.default.parallelism</td>
<td>8</td> <td>
<ul>
<li>Mesos fine grained mode: 8</li>
<li>Local mode: core number of the local machine</li>
<li>Others: total core number of all executor nodes or 2, whichever is larger</li>
</ul>
</td>
<td> <td>
Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>, Default number of tasks to use across the cluster for distributed shuffle operations (<code>groupByKey</code>,
<code>reduceByKey</code>, etc) when not set by user. <code>reduceByKey</code>, etc) when not set by user.
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment