-
- Downloads
[SPARK-17648][CORE] TaskScheduler really needs offers to be an IndexedSeq
## What changes were proposed in this pull request? The Seq[WorkerOffer] is accessed by index, so it really should be an IndexedSeq, otherwise an O(n) operation becomes O(n^2). In practice this hasn't been an issue b/c where these offers are generated, the call to `.toSeq` just happens to create an IndexedSeq anyway.I got bitten by this in performance tests I was doing, and its better for the types to be more precise so eg. a change in Scala doesn't destroy performance. ## How was this patch tested? Unit tests via jenkins. Author: Imran Rashid <irashid@cloudera.com> Closes #15221 from squito/SPARK-17648.
Showing
- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala 2 additions, 2 deletions.../scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
- core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala 2 additions, 2 deletions...ark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala
- core/src/main/scala/org/apache/spark/scheduler/local/LocalSchedulerBackend.scala 1 addition, 1 deletion.../apache/spark/scheduler/local/LocalSchedulerBackend.scala
- core/src/test/scala/org/apache/spark/scheduler/SchedulerIntegrationSuite.scala 3 additions, 4 deletions...rg/apache/spark/scheduler/SchedulerIntegrationSuite.scala
- core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala 16 additions, 16 deletions...a/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala
- mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala 1 addition, 1 deletion...uler/cluster/mesos/MesosFineGrainedSchedulerBackend.scala
- mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala 1 addition, 1 deletion...cluster/mesos/MesosFineGrainedSchedulerBackendSuite.scala
Loading
Please register or sign in to comment