Skip to content
Snippets Groups Projects
Commit 200f01c8 authored by Marcelo Vanzin's avatar Marcelo Vanzin
Browse files

[SPARK-15760][DOCS] Add documentation for package-related configs.

While there, also document spark.files and spark.jars. Text is the
same as the spark-submit help text with some minor adjustments.

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #13502 from vanzin/SPARK-15760.
parent 3ec4461c
No related branches found
No related tags found
No related merge requests found
......@@ -380,6 +380,53 @@ Apart from these, the following properties are also available, and may be useful
from JVM to Python worker for every task.
</td>
</tr>
<tr>
<td><code>spark.files</code></td>
<td></td>
<td>
Comma-separated list of files to be placed in the working directory of each executor.
</td>
</tr>
<tr>
<td><code>spark.submit.pyFiles</code></td>
<td></td>
<td>
Comma-separated list of .zip, .egg, or .py files to place on the PYTHONPATH for Python apps.
</td>
</tr>
<tr>
<td><code>spark.jars</code></td>
<td></td>
<td>
Comma-separated list of local jars to include on the driver and executor classpaths.
</td>
</tr>
<tr>
<td><code>spark.jars.packages</code></td>
<td></td>
<td>
Comma-separated list of maven coordinates of jars to include on the driver and executor
classpaths. Will search the local maven repo, then maven central and any additional remote
repositories given by <code>spark.jars.ivy</code>. The format for the coordinates should be
groupId:artifactId:version.
</td>
</tr>
<tr>
<td><code>spark.jars.excludes</code></td>
<td></td>
<td>
Comma-separated list of groupId:artifactId, to exclude while resolving the dependencies
provided in <code>spark.jars.packages</code> to avoid dependency conflicts.
</td>
</tr>
<tr>
<td><code>spark.jars.ivy</code></td>
<td></td>
<td>
Comma-separated list of additional remote repositories to search for the coordinates given
with <code>spark.jars.packages</code>.
</td>
</tr>
</table>
#### Shuffle Behavior
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment