Skip to content
Snippets Groups Projects
  • Davies Liu's avatar
    e2c901b4
    [SPARK-2871] [PySpark] add countApproxDistinct() API · e2c901b4
    Davies Liu authored
    RDD.countApproxDistinct(relativeSD=0.05):
    
            :: Experimental ::
            Return approximate number of distinct elements in the RDD.
    
            The algorithm used is based on streamlib's implementation of
            "HyperLogLog in Practice: Algorithmic Engineering of a State
            of The Art Cardinality Estimation Algorithm", available
            <a href="http://dx.doi.org/10.1145/2452376.2452456">here</a>.
    
            This support all the types of objects, which is supported by
            Pyrolite, nearly all builtin types.
    
            param relativeSD Relative accuracy. Smaller values create
                               counters that require more space.
                               It must be greater than 0.000017.
    
            >>> n = sc.parallelize(range(1000)).map(str).countApproxDistinct()
            >>> 950 < n < 1050
            True
            >>> n = sc.parallelize([i % 20 for i in range(1000)]).countApproxDistinct()
            >>> 18 < n < 22
            True
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2142 from davies/countApproxDistinct and squashes the following commits:
    
    e20da47 [Davies Liu] remove the correction in Python
    c38c4e4 [Davies Liu] fix doc tests
    2ab157c [Davies Liu] fix doc tests
    9d2565f [Davies Liu] add commments and link for hash collision correction
    d306492 [Davies Liu] change range of hash of tuple to [0, maxint]
    ded624f [Davies Liu] calculate hash in Python
    4cba98f [Davies Liu] add more tests
    a85a8c6 [Davies Liu] Merge branch 'master' into countApproxDistinct
    e97e342 [Davies Liu] add countApproxDistinct()
    e2c901b4
    History
    [SPARK-2871] [PySpark] add countApproxDistinct() API
    Davies Liu authored
    RDD.countApproxDistinct(relativeSD=0.05):
    
            :: Experimental ::
            Return approximate number of distinct elements in the RDD.
    
            The algorithm used is based on streamlib's implementation of
            "HyperLogLog in Practice: Algorithmic Engineering of a State
            of The Art Cardinality Estimation Algorithm", available
            <a href="http://dx.doi.org/10.1145/2452376.2452456">here</a>.
    
            This support all the types of objects, which is supported by
            Pyrolite, nearly all builtin types.
    
            param relativeSD Relative accuracy. Smaller values create
                               counters that require more space.
                               It must be greater than 0.000017.
    
            >>> n = sc.parallelize(range(1000)).map(str).countApproxDistinct()
            >>> 950 < n < 1050
            True
            >>> n = sc.parallelize([i % 20 for i in range(1000)]).countApproxDistinct()
            >>> 18 < n < 22
            True
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #2142 from davies/countApproxDistinct and squashes the following commits:
    
    e20da47 [Davies Liu] remove the correction in Python
    c38c4e4 [Davies Liu] fix doc tests
    2ab157c [Davies Liu] fix doc tests
    9d2565f [Davies Liu] add commments and link for hash collision correction
    d306492 [Davies Liu] change range of hash of tuple to [0, maxint]
    ded624f [Davies Liu] calculate hash in Python
    4cba98f [Davies Liu] add more tests
    a85a8c6 [Davies Liu] Merge branch 'master' into countApproxDistinct
    e97e342 [Davies Liu] add countApproxDistinct()
tests.py 50.99 KiB