Skip to content
  • Davies Liu's avatar
    434bea1c
    [SPARK-2983] [PySpark] improve performance of sortByKey() · 434bea1c
    Davies Liu authored
    1. skip partitionBy() when numOfPartition is 1
    2. use bisect_left (O(lg(N))) instread of loop (O(N)) in
    rangePartitioner
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1898 from davies/sort and squashes the following commits:
    
    0a9608b [Davies Liu] Merge branch 'master' into sort
    1cf9565 [Davies Liu] improve performance of sortByKey()
    434bea1c
    [SPARK-2983] [PySpark] improve performance of sortByKey()
    Davies Liu authored
    1. skip partitionBy() when numOfPartition is 1
    2. use bisect_left (O(lg(N))) instread of loop (O(N)) in
    rangePartitioner
    
    Author: Davies Liu <davies.liu@gmail.com>
    
    Closes #1898 from davies/sort and squashes the following commits:
    
    0a9608b [Davies Liu] Merge branch 'master' into sort
    1cf9565 [Davies Liu] improve performance of sortByKey()
Loading