-
- Downloads
[SPARK-21176][WEB UI] Use a single ProxyServlet to proxy all workers and applications
## What changes were proposed in this pull request? Currently, each application and each worker creates their own proxy servlet. Each proxy servlet is backed by its own HTTP client and a relatively large number of selector threads. This is excessive but was fixed (to an extent) by https://github.com/apache/spark/pull/18437. However, a single HTTP client (backed by a single selector thread) should be enough to handle all proxy requests. This PR creates a single proxy servlet no matter how many applications and workers there are. ## How was this patch tested? . The unit tests for rewriting proxied locations and headers were updated. I then spun up a 100 node cluster to ensure that proxy'ing worked correctly jiangxb1987 Please let me know if there's anything else I can do to help push this thru. Thanks! Author: Anderson Osagie <osagie@gmail.com> Closes #18499 from aosagie/fix/minimize-proxy-threads.
Showing
- core/src/main/scala/org/apache/spark/deploy/master/Master.scala 3 additions, 12 deletions...rc/main/scala/org/apache/spark/deploy/master/Master.scala
- core/src/main/scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala 10 additions, 11 deletions...scala/org/apache/spark/deploy/master/ui/MasterWebUI.scala
- core/src/main/scala/org/apache/spark/ui/JettyUtils.scala 24 additions, 21 deletionscore/src/main/scala/org/apache/spark/ui/JettyUtils.scala
- core/src/test/scala/org/apache/spark/ui/UISuite.scala 9 additions, 11 deletionscore/src/test/scala/org/apache/spark/ui/UISuite.scala
Loading
Please register or sign in to comment