-
- Downloads
[SPARK-15891][YARN] Clean up some logging in the YARN AM.
To make the log file more readable, rework some of the logging done by the AM: - log executor command / env just once, since they're all almost the same; the information that changes, such as executor ID, is already available in other log messages. - avoid printing logs when nothing happens, especially when updating the container requests in the allocator. - print fewer log messages when requesting many unlocalized executors, instead of repeating the same message multiple times. - removed some logs that seemed unnecessary. In the process, I slightly fixed up the wording in a few log messages, and did some minor clean up of method arguments that were redundant. Tested by running existing unit tests, and analyzing the logs of an application that exercises dynamic allocation by forcing executors to be allocated and be killed in waves. Author: Marcelo Vanzin <vanzin@cloudera.com> Closes #14943 from vanzin/SPARK-15891.
Showing
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala 13 additions, 2 deletions...cala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala 45 additions, 47 deletions...scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
- yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala 22 additions, 12 deletions...in/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
- yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala 2 additions, 2 deletions...ala/org/apache/spark/deploy/yarn/YarnAllocatorSuite.scala
Loading
Please register or sign in to comment