Skip to content
Snippets Groups Projects
Commit fdf63f12 authored by Kevin Conor's avatar Kevin Conor Committed by Davies Liu
Browse files

[SPARK-8339] [PYSPARK] integer division for python 3

Itertools islice requires an integer for the stop argument.  Switching to integer division here prevents a ValueError when vs is evaluated above.

davies

This is my original work, and I license it to the project.

Author: Kevin Conor <kevin@discoverybayconsulting.com>

Closes #6794 from kconor/kconor-patch-1 and squashes the following commits:

da5e700 [Kevin Conor] Integer division for batch size
parent a2016b4b
No related branches found
No related tags found
No related merge requests found
......@@ -272,7 +272,7 @@ class AutoBatchedSerializer(BatchedSerializer):
if size < best:
batch *= 2
elif size > best * 10 and batch > 1:
batch /= 2
batch //= 2
def __repr__(self):
return "AutoBatchedSerializer(%s)" % self.serializer
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment