Skip to content
Snippets Groups Projects
  1. Jan 06, 2020
  2. Jul 11, 2019
  3. Jun 16, 2019
  4. Apr 04, 2019
  5. Jan 02, 2019
  6. Oct 03, 2018
  7. Sep 17, 2018
    • Alex Ellis (VMware)'s avatar
      Alter graceful shutdown sequence · e67811c9
      Alex Ellis (VMware) authored
      
      - the shutdown sequence meant that the kubelet was still passing
      work to the watchdog after the HTTP socket was closed. This change
      means that the kubelet has a chance to run its check before we
      finally stop accepting new connections. It will require some
      basic co-ordination between the kubelet's checking period and the
      "write_timeout" value in the container.
      
      Tested with Kubernetes on GKE - before the change some Pods were
      giving a connection refused error due to them being not detected
      as unhealthy. Now I receive 0% error rate even with 20 qps.
      
      Issue was shown by scaling to 20 replicas, starting a test with
      hey and then scaling to 1 replica while tailing the logs from the
      gateway. Before I saw some 502, now I see just 200s.
      
      Signed-off-by: default avatarAlex Ellis (VMware) <alexellis2@gmail.com>
      e67811c9
  8. Aug 22, 2018
  9. Aug 13, 2018
  10. Apr 14, 2018
  11. Mar 24, 2018
    • Alex Ellis (VMware)'s avatar
      Add feature for graceful shutdown of HTTP server · de2c74fc
      Alex Ellis (VMware) authored
      If the watchdog is sent SIGTERM from an external process then it
      should stop accepting new connections and attempt to finish the
      work in progress. This change makes use of the new ability in Go
      1.9 and onwards to cancel a HTTP server gracefully.
      
      The write_timeout duration is used as a grace period to allow all
      in-flight requests to complete. The pattern is taken directly from
      the offical example in the Golang documentation. [1]
      
      Further tuning and testing may be needed for Windows containers which
      have a different set of signals for closing work. This change aims
      to cover the majority use-case for Linux containers.
      
      The HTTP health-check is also invalidated by creating an and
      expression with the existing lock file.
      
      Tested with Kubernetes by deploying a custom watchdog and the
      fprocess of `env`. Log message was observed when scaling down and
      connections stopped being accepted on terminating replica.
      
      Also corrects some typos from previous PR.
      
      [1] https://golang.org/pkg/net/http/#Server.Shutdown
      
      
      
      Signed-off-by: default avatarAlex Ellis (VMware) <alexellis2@gmail.com>
      de2c74fc
  12. Mar 23, 2018
  13. Mar 20, 2018
  14. Mar 17, 2018
  15. Nov 06, 2017
  16. Nov 04, 2017
  17. Oct 31, 2017
  18. Oct 29, 2017
  19. Oct 23, 2017
  20. Oct 11, 2017
  21. Oct 04, 2017
  22. Oct 01, 2017
  23. Sep 09, 2017
  24. Aug 23, 2017
  25. Jul 19, 2017
  26. Jul 07, 2017
    • Alex Ellis's avatar
      Watchdog refurbishments · 4d058967
      Alex Ellis authored
      
      - Watchdog - allow new methods with and without body.
      - Enforce hard-timeout via exec_timeout variable.
      - Correct bug in timeouts for read/write of HTTP.
      - Documentation for new verbs and hard timeout.
      
      Signed-off-by: default avatarAlex Ellis <alexellis2@gmail.com>
      4d058967
  27. May 09, 2017
  28. May 03, 2017
  29. Apr 27, 2017
  30. Apr 10, 2017
  31. Apr 05, 2017
Loading