Skip to content
  • CrazyJvm's avatar
    b520b645
    SPARK-2400 : fix spark.yarn.max.executor.failures explaination · b520b645
    CrazyJvm authored
    According to
    ```scala
      private val maxNumExecutorFailures = sparkConf.getInt("spark.yarn.max.executor.failures",
        sparkConf.getInt("spark.yarn.max.worker.failures", math.max(args.numExecutors * 2, 3)))
    ```
    default value should be numExecutors * 2, with minimum of 3,  and it's same to the config
    `spark.yarn.max.worker.failures`
    
    Author: CrazyJvm <crazyjvm@gmail.com>
    
    Closes #1282 from CrazyJvm/yarn-doc and squashes the following commits:
    
    1a5f25b [CrazyJvm] remove deprecated config
    c438aec [CrazyJvm] fix style
    86effa6 [CrazyJvm] change expression
    211f130 [CrazyJvm] fix html tag
    2900d23 [CrazyJvm] fix style
    a4b2e27 [CrazyJvm] fix configuration spark.yarn.max.executor.failures
    b520b645
    SPARK-2400 : fix spark.yarn.max.executor.failures explaination
    CrazyJvm authored
    According to
    ```scala
      private val maxNumExecutorFailures = sparkConf.getInt("spark.yarn.max.executor.failures",
        sparkConf.getInt("spark.yarn.max.worker.failures", math.max(args.numExecutors * 2, 3)))
    ```
    default value should be numExecutors * 2, with minimum of 3,  and it's same to the config
    `spark.yarn.max.worker.failures`
    
    Author: CrazyJvm <crazyjvm@gmail.com>
    
    Closes #1282 from CrazyJvm/yarn-doc and squashes the following commits:
    
    1a5f25b [CrazyJvm] remove deprecated config
    c438aec [CrazyJvm] fix style
    86effa6 [CrazyJvm] change expression
    211f130 [CrazyJvm] fix html tag
    2900d23 [CrazyJvm] fix style
    a4b2e27 [CrazyJvm] fix configuration spark.yarn.max.executor.failures
Loading