Marcelo Vanzin
authored
Currently the code monitoring the launch of the client AM uses the value of spark.yarn.report.interval as the interval for polling the RM; if someone has that value to a really large interval, it would take that long to detect that the client AM has started, which is not expected. Instead, have a separate config for the interval to use when the client AM is starting. The other config is still used in cluster mode, and to detect the status of the client AM after it is already running. Tested by running client and cluster mode apps with a modified value of spark.yarn.report.interval, verifying client AM launch is detected before that interval elapses. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #18380 from vanzin/SPARK-16019.