Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    4a377aff
    [SPARK-3721] [PySpark] broadcast objects larger than 2G · 4a377aff
    Davies Liu authored
    This patch will bring support for broadcasting objects larger than 2G.
    
    pickle, zlib, FrameSerializer and Array[Byte] all can not support objects larger than 2G, so this patch introduce LargeObjectSerializer to serialize broadcast objects, the object will be serialized and compressed into small chunks, it also change the type of Broadcast[Array[Byte]]] into Broadcast[Array[Array[Byte]]]].
    
    Testing for support broadcast objects larger than 2G is slow and memory hungry, so this is tested manually, could be added into SparkPerf.
    
    Author: Davies Liu <davies@databricks.com>
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2659 from davies/huge and squashes the following commits:
    
    7b57a14 [Davies Liu] add more tests for broadcast
    28acff9 [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    a2f6a02 [Davies Liu] bug fix
    4820613 [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    5875c73 [Davies Liu] address comments
    10a349b [Davies Liu] address comments
    0c33016 [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    6182c8f [Davies Liu] Merge branch 'master' into huge
    d94b68f [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    2514848 [Davies Liu] address comments
    fda395b [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    1c2d928 [Davies Liu] fix scala style
    091b107 [Davies Liu] broadcast objects larger than 2G
    4a377aff
    History
    [SPARK-3721] [PySpark] broadcast objects larger than 2G
    Davies Liu authored
    This patch will bring support for broadcasting objects larger than 2G.
    
    pickle, zlib, FrameSerializer and Array[Byte] all can not support objects larger than 2G, so this patch introduce LargeObjectSerializer to serialize broadcast objects, the object will be serialized and compressed into small chunks, it also change the type of Broadcast[Array[Byte]]] into Broadcast[Array[Array[Byte]]]].
    
    Testing for support broadcast objects larger than 2G is slow and memory hungry, so this is tested manually, could be added into SparkPerf.
    
    Author: Davies Liu <davies@databricks.com>
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2659 from davies/huge and squashes the following commits:
    
    7b57a14 [Davies Liu] add more tests for broadcast
    28acff9 [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    a2f6a02 [Davies Liu] bug fix
    4820613 [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    5875c73 [Davies Liu] address comments
    10a349b [Davies Liu] address comments
    0c33016 [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    6182c8f [Davies Liu] Merge branch 'master' into huge
    d94b68f [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    2514848 [Davies Liu] address comments
    fda395b [Davies Liu] Merge branch 'master' of github.com:apache/spark into huge
    1c2d928 [Davies Liu] fix scala style
    091b107 [Davies Liu] broadcast objects larger than 2G
broadcast.py 2.78 KiB
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

"""
>>> from pyspark.context import SparkContext
>>> sc = SparkContext('local', 'test')
>>> b = sc.broadcast([1, 2, 3, 4, 5])
>>> b.value
[1, 2, 3, 4, 5]
>>> sc.parallelize([0, 0]).flatMap(lambda x: b.value).collect()
[1, 2, 3, 4, 5, 1, 2, 3, 4, 5]
>>> b.unpersist()

>>> large_broadcast = sc.broadcast(list(range(10000)))
"""
import os

from pyspark.serializers import LargeObjectSerializer


__all__ = ['Broadcast']


# Holds broadcasted data received from Java, keyed by its id.
_broadcastRegistry = {}


def _from_id(bid):
    from pyspark.broadcast import _broadcastRegistry
    if bid not in _broadcastRegistry:
        raise Exception("Broadcast variable '%s' not loaded!" % bid)
    return _broadcastRegistry[bid]


class Broadcast(object):

    """
    A broadcast variable created with
    L{SparkContext.broadcast()<pyspark.context.SparkContext.broadcast>}.
    Access its value through C{.value}.
    """

    def __init__(self, bid, value, java_broadcast=None,
                 pickle_registry=None, path=None):
        """
        Should not be called directly by users -- use
        L{SparkContext.broadcast()<pyspark.context.SparkContext.broadcast>}
        instead.
        """
        self.bid = bid
        if path is None:
            self._value = value
        self._jbroadcast = java_broadcast
        self._pickle_registry = pickle_registry
        self.path = path

    @property
    def value(self):
        """ Return the broadcasted value
        """
        if not hasattr(self, "_value") and self.path is not None:
            ser = LargeObjectSerializer()
            self._value = ser.load_stream(open(self.path)).next()
        return self._value

    def unpersist(self, blocking=False):
        """
        Delete cached copies of this broadcast on the executors.
        """
        self._jbroadcast.unpersist(blocking)
        os.unlink(self.path)

    def __reduce__(self):
        self._pickle_registry.add(self)
        return (_from_id, (self.bid, ))


if __name__ == "__main__":
    import doctest
    doctest.testmod()