Skip to content
  • Luc Bourlier's avatar
    0bb5b733
    [SPARK-13002][MESOS] Send initial request of executors for dyn allocation · 0bb5b733
    Luc Bourlier authored
    Fix for [SPARK-13002](https://issues.apache.org/jira/browse/SPARK-13002) about the initial number of executors when running with dynamic allocation on Mesos.
    Instead of fixing it just for the Mesos case, made the change in `ExecutorAllocationManager`. It is already driving the number of executors running on Mesos, only no the initial value.
    
    The `None` and `Some(0)` are internal details on the computation of resources to reserved, in the Mesos backend scheduler. `executorLimitOption` has to be initialized correctly, otherwise the Mesos backend scheduler will, either, create to many executors at launch, or not create any executors and not be able to recover from this state.
    
    Removed the 'special case' description in the doc. It was not totally accurate, and is not needed anymore.
    
    This doesn't fix the same problem visible with Spark standalone. There is no straightforward way to send the initial value in standalone mode.
    
    Somebody knowing this part of the yarn support should review this change.
    
    Author: Luc Bourlier <luc.bourlier@typesafe.com>
    
    Closes #11047 from skyluc/issue/initial-dyn-alloc-2.
    0bb5b733
    [SPARK-13002][MESOS] Send initial request of executors for dyn allocation
    Luc Bourlier authored
    Fix for [SPARK-13002](https://issues.apache.org/jira/browse/SPARK-13002) about the initial number of executors when running with dynamic allocation on Mesos.
    Instead of fixing it just for the Mesos case, made the change in `ExecutorAllocationManager`. It is already driving the number of executors running on Mesos, only no the initial value.
    
    The `None` and `Some(0)` are internal details on the computation of resources to reserved, in the Mesos backend scheduler. `executorLimitOption` has to be initialized correctly, otherwise the Mesos backend scheduler will, either, create to many executors at launch, or not create any executors and not be able to recover from this state.
    
    Removed the 'special case' description in the doc. It was not totally accurate, and is not needed anymore.
    
    This doesn't fix the same problem visible with Spark standalone. There is no straightforward way to send the initial value in standalone mode.
    
    Somebody knowing this part of the yarn support should review this change.
    
    Author: Luc Bourlier <luc.bourlier@typesafe.com>
    
    Closes #11047 from skyluc/issue/initial-dyn-alloc-2.
Loading