-
- Downloads
[SPARK-8425][CORE] Application Level Blacklisting
## What changes were proposed in this pull request? This builds upon the blacklisting introduced in SPARK-17675 to add blacklisting of executors and nodes for an entire Spark application. Resources are blacklisted based on tasks that fail, in tasksets that eventually complete successfully; they are automatically returned to the pool of active resources based on a timeout. Full details are available in a design doc attached to the jira. ## How was this patch tested? Added unit tests, ran them via Jenkins, also ran a handful of them in a loop to check for flakiness. The added tests include: - verifying BlacklistTracker works correctly - verifying TaskSchedulerImpl interacts with BlacklistTracker correctly (via a mock BlacklistTracker) - an integration test for the entire scheduler with blacklisting in a few different scenarios Author: Imran Rashid <irashid@cloudera.com> Author: mwws <wei.mao@intel.com> Closes #14079 from squito/blacklist-SPARK-8425.
Showing
- core/src/main/scala/org/apache/spark/internal/config/package.scala 10 additions, 0 deletions...main/scala/org/apache/spark/internal/config/package.scala
- core/src/main/scala/org/apache/spark/scheduler/BlacklistTracker.scala 270 additions, 2 deletions...n/scala/org/apache/spark/scheduler/BlacklistTracker.scala
- core/src/main/scala/org/apache/spark/scheduler/ExecutorFailuresInTaskSet.scala 12 additions, 8 deletions...rg/apache/spark/scheduler/ExecutorFailuresInTaskSet.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 51 additions, 4 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala 15 additions, 9 deletions...n/scala/org/apache/spark/scheduler/TaskSetBlacklist.scala
- core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala 14 additions, 6 deletions...ain/scala/org/apache/spark/scheduler/TaskSetManager.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedClusterMessage.scala 2 additions, 1 deletion...spark/scheduler/cluster/CoarseGrainedClusterMessage.scala
- core/src/test/scala/org/apache/spark/HeartbeatReceiverSuite.scala 2 additions, 2 deletions.../test/scala/org/apache/spark/HeartbeatReceiverSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/BlacklistTrackerSuite.scala 350 additions, 2 deletions...la/org/apache/spark/scheduler/BlacklistTrackerSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala 106 additions, 3 deletions...a/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetBlacklistSuite.scala 3 additions, 3 deletions...la/org/apache/spark/scheduler/TaskSetBlacklistSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSetManagerSuite.scala 26 additions, 23 deletions...cala/org/apache/spark/scheduler/TaskSetManagerSuite.scala
- docs/configuration.md 30 additions, 0 deletionsdocs/configuration.md
- resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 3 additions, 3 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
- resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala 20 additions, 1 deletion...in/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
- resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala 10 additions, 2 deletions...apache/spark/scheduler/cluster/YarnSchedulerBackend.scala
- resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala 26 additions, 7 deletions...ala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
- yarn/src/test/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackendSuite.scala 57 additions, 0 deletions...e/spark/scheduler/cluster/YarnSchedulerBackendSuite.scala
Loading
Please register or sign in to comment