Skip to content
Snippets Groups Projects
  • Ryan Blue's avatar
    738f134b
    [SPARK-13723][YARN] Change behavior of --num-executors with dynamic allocation. · 738f134b
    Ryan Blue authored
    ## What changes were proposed in this pull request?
    
    This changes the behavior of --num-executors and spark.executor.instances when using dynamic allocation. Instead of turning dynamic allocation off, it uses the value for the initial number of executors.
    
    This changes was discussed on [SPARK-13723](https://issues.apache.org/jira/browse/SPARK-13723). I highly recommend using it while we can change the behavior for 2.0.0. In practice, the 1.x behavior causes unexpected behavior for users (it is not clear that it disables dynamic allocation) and wastes cluster resources because users rarely notice the log message.
    
    ## How was this patch tested?
    
    This patch updates tests and adds a test for Utils.getDynamicAllocationInitialExecutors.
    
    Author: Ryan Blue <blue@apache.org>
    
    Closes #13338 from rdblue/SPARK-13723-num-executors-with-dynamic-allocation.
    738f134b
    History
    [SPARK-13723][YARN] Change behavior of --num-executors with dynamic allocation.
    Ryan Blue authored
    ## What changes were proposed in this pull request?
    
    This changes the behavior of --num-executors and spark.executor.instances when using dynamic allocation. Instead of turning dynamic allocation off, it uses the value for the initial number of executors.
    
    This changes was discussed on [SPARK-13723](https://issues.apache.org/jira/browse/SPARK-13723). I highly recommend using it while we can change the behavior for 2.0.0. In practice, the 1.x behavior causes unexpected behavior for users (it is not clear that it disables dynamic allocation) and wastes cluster resources because users rarely notice the log message.
    
    ## How was this patch tested?
    
    This patch updates tests and adds a test for Utils.getDynamicAllocationInitialExecutors.
    
    Author: Ryan Blue <blue@apache.org>
    
    Closes #13338 from rdblue/SPARK-13723-num-executors-with-dynamic-allocation.