-
- Downloads
[SPARK-9024] Unsafe HashJoin/HashOuterJoin/HashSemiJoin
This PR introduce unsafe version (using UnsafeRow) of HashJoin, HashOuterJoin and HashSemiJoin, including the broadcast one and shuffle one (except FullOuterJoin, which is better to be implemented using SortMergeJoin). It use HashMap to store UnsafeRow right now, will change to use BytesToBytesMap for better performance (in another PR). Author: Davies Liu <davies@databricks.com> Closes #7480 from davies/unsafe_join and squashes the following commits: 6294b1e [Davies Liu] fix projection 10583f1 [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join dede020 [Davies Liu] fix test 84c9807 [Davies Liu] address comments a05b4f6 [Davies Liu] support UnsafeRow in LeftSemiJoinBNL and BroadcastNestedLoopJoin 611d2ed [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join 9481ae8 [Davies Liu] return UnsafeRow after join() ca2b40f [Davies Liu] revert unrelated change 68f5cd9 [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join 0f4380d [Davies Liu] ada a comment 69e38f5 [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join 1a40f02 [Davies Liu] refactor ab1690f [Davies Liu] address comments 60371f2 [Davies Liu] use UnsafeRow in SemiJoin a6c0b7d [Davies Liu] Merge branch 'master' of github.com:apache/spark into unsafe_join 184b852 [Davies Liu] fix style 6acbb11 [Davies Liu] fix tests 95d0762 [Davies Liu] remove println bea4a50 [Davies Liu] Unsafe HashJoin
Showing
- sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java 47 additions, 3 deletions.../org/apache/spark/sql/catalyst/expressions/UnsafeRow.java
- sql/catalyst/src/main/java/org/apache/spark/sql/execution/UnsafeExternalRowSorter.java 2 additions, 8 deletions...g/apache/spark/sql/execution/UnsafeExternalRowSorter.java
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BoundAttribute.scala 17 additions, 2 deletions...pache/spark/sql/catalyst/expressions/BoundAttribute.scala
- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Projection.scala 33 additions, 1 deletion...rg/apache/spark/sql/catalyst/expressions/Projection.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoin.scala 1 addition, 1 deletion.../apache/spark/sql/execution/joins/BroadcastHashJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashOuterJoin.scala 7 additions, 25 deletions...he/spark/sql/execution/joins/BroadcastHashOuterJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastLeftSemiJoinHash.scala 2 additions, 3 deletions...spark/sql/execution/joins/BroadcastLeftSemiJoinHash.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoin.scala 25 additions, 12 deletions...e/spark/sql/execution/joins/BroadcastNestedLoopJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala 35 additions, 8 deletions...scala/org/apache/spark/sql/execution/joins/HashJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashOuterJoin.scala 71 additions, 11 deletions.../org/apache/spark/sql/execution/joins/HashOuterJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashSemiJoin.scala 47 additions, 27 deletions...a/org/apache/spark/sql/execution/joins/HashSemiJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala 81 additions, 4 deletions...org/apache/spark/sql/execution/joins/HashedRelation.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/LeftSemiJoinBNL.scala 3 additions, 0 deletions...rg/apache/spark/sql/execution/joins/LeftSemiJoinBNL.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/LeftSemiJoinHash.scala 2 additions, 2 deletions...g/apache/spark/sql/execution/joins/LeftSemiJoinHash.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoin.scala 1 addition, 1 deletion...g/apache/spark/sql/execution/joins/ShuffledHashJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashOuterJoin.scala 7 additions, 6 deletions...che/spark/sql/execution/joins/ShuffledHashOuterJoin.scala
- sql/core/src/main/scala/org/apache/spark/sql/execution/rowFormatConverters.scala 16 additions, 5 deletions.../org/apache/spark/sql/execution/rowFormatConverters.scala
- sql/core/src/test/scala/org/apache/spark/sql/UnsafeRowSuite.scala 2 additions, 2 deletions.../src/test/scala/org/apache/spark/sql/UnsafeRowSuite.scala
- sql/core/src/test/scala/org/apache/spark/sql/execution/joins/HashedRelationSuite.scala 38 additions, 11 deletions...pache/spark/sql/execution/joins/HashedRelationSuite.scala
- unsafe/src/main/java/org/apache/spark/unsafe/hash/Murmur3_x86_32.java 7 additions, 3 deletions...ain/java/org/apache/spark/unsafe/hash/Murmur3_x86_32.java
Loading
Please register or sign in to comment