-
- Downloads
[SPARK-14731][shuffle]Revert SPARK-12130 to make 2.0 shuffle service compatible with 1.x
## What changes were proposed in this pull request? SPARK-12130 make 2.0 shuffle service incompatible with 1.x. So from discussion: [http://apache-spark-developers-list.1001551.n3.nabble.com/YARN-Shuffle-service-and-its-compatibility-td17222.html](url) we should maintain compatibility between Spark 1.x and Spark 2.x's shuffle service. I put string comparison into executor's register at first avoid string comparison in getBlockData every time. ## How was this patch tested? N/A Author: Lianhui Wang <lianhuiwang09@gmail.com> Closes #12568 from lianhuiwang/SPARK-14731.
Showing
- common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java 9 additions, 6 deletions...e/spark/network/shuffle/ExternalShuffleBlockResolver.java
- common/network-shuffle/src/test/java/org/apache/spark/network/sasl/SaslIntegrationSuite.java 2 additions, 1 deletion...a/org/apache/spark/network/sasl/SaslIntegrationSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolverSuite.java 6 additions, 5 deletions...rk/network/shuffle/ExternalShuffleBlockResolverSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleCleanupSuite.java 8 additions, 7 deletions...he/spark/network/shuffle/ExternalShuffleCleanupSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleIntegrationSuite.java 4 additions, 7 deletions...park/network/shuffle/ExternalShuffleIntegrationSuite.java
- common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/ExternalShuffleSecuritySuite.java 2 additions, 1 deletion...e/spark/network/shuffle/ExternalShuffleSecuritySuite.java
- core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala 0 additions, 3 deletions.../main/scala/org/apache/spark/shuffle/ShuffleManager.scala
- core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala 0 additions, 2 deletions...la/org/apache/spark/shuffle/sort/SortShuffleManager.scala
- core/src/main/scala/org/apache/spark/storage/BlockManager.scala 1 addition, 1 deletion...rc/main/scala/org/apache/spark/storage/BlockManager.scala
- yarn/src/test/scala/org/apache/spark/network/yarn/YarnShuffleServiceSuite.scala 7 additions, 6 deletions...g/apache/spark/network/yarn/YarnShuffleServiceSuite.scala
Loading
Please register or sign in to comment