Skip to content
Snippets Groups Projects
Unverified Commit 28ab0ec4 authored by Yuexin Zhang's avatar Yuexin Zhang Committed by Sean Owen
Browse files

[SPARK-19006][DOCS] mention spark.kryoserializer.buffer.max must be less than 2048m in doc

## What changes were proposed in this pull request?

On configuration doc page:https://spark.apache.org/docs/latest/configuration.html
We mentioned spark.kryoserializer.buffer.max : Maximum allowable size of Kryo serialization buffer. This must be larger than any object you attempt to serialize. Increase this if you get a "buffer limit exceeded" exception inside Kryo.
from source code, it has hard coded upper limit :
```
val maxBufferSizeMb = conf.getSizeAsMb("spark.kryoserializer.buffer.max", "64m").toInt
if (maxBufferSizeMb >= ByteUnit.GiB.toMiB(2))
{ throw new IllegalArgumentException("spark.kryoserializer.buffer.max must be less than " + s"2048 mb, got: + $maxBufferSizeMb mb.") }
```
We should mention "this value must be less than 2048 mb" on the configuration doc page as well.

## How was this patch tested?

None. Since it's minor doc change.

Author: Yuexin Zhang <yxzhang@cloudera.com>

Closes #16412 from cnZach/SPARK-19006.
parent d8e14db8
No related branches found
No related tags found
No related merge requests found
...@@ -840,8 +840,8 @@ Apart from these, the following properties are also available, and may be useful ...@@ -840,8 +840,8 @@ Apart from these, the following properties are also available, and may be useful
<td>64m</td> <td>64m</td>
<td> <td>
Maximum allowable size of Kryo serialization buffer. This must be larger than any Maximum allowable size of Kryo serialization buffer. This must be larger than any
object you attempt to serialize. Increase this if you get a "buffer limit exceeded" exception object you attempt to serialize and must be less than 2048m.
inside Kryo. Increase this if you get a "buffer limit exceeded" exception inside Kryo.
</td> </td>
</tr> </tr>
<tr> <tr>
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment