Skip to content
Snippets Groups Projects
  1. Oct 19, 2018
  2. Oct 03, 2018
  3. Sep 19, 2018
  4. Sep 17, 2018
    • Alex Ellis (VMware)'s avatar
      Alter graceful shutdown sequence · e67811c9
      Alex Ellis (VMware) authored
      
      - the shutdown sequence meant that the kubelet was still passing
      work to the watchdog after the HTTP socket was closed. This change
      means that the kubelet has a chance to run its check before we
      finally stop accepting new connections. It will require some
      basic co-ordination between the kubelet's checking period and the
      "write_timeout" value in the container.
      
      Tested with Kubernetes on GKE - before the change some Pods were
      giving a connection refused error due to them being not detected
      as unhealthy. Now I receive 0% error rate even with 20 qps.
      
      Issue was shown by scaling to 20 replicas, starting a test with
      hey and then scaling to 1 replica while tailing the logs from the
      gateway. Before I saw some 502, now I see just 200s.
      
      Signed-off-by: default avatarAlex Ellis (VMware) <alexellis2@gmail.com>
      e67811c9
  5. Sep 06, 2018
  6. Aug 22, 2018
  7. Aug 21, 2018
  8. Aug 13, 2018
  9. Jul 10, 2018
  10. Jul 09, 2018
  11. Jul 06, 2018
  12. Jun 16, 2018
  13. May 11, 2018
  14. Apr 14, 2018
  15. Apr 13, 2018
  16. Apr 03, 2018
  17. Mar 24, 2018
    • Alex Ellis (VMware)'s avatar
      Add feature for graceful shutdown of HTTP server · de2c74fc
      Alex Ellis (VMware) authored
      If the watchdog is sent SIGTERM from an external process then it
      should stop accepting new connections and attempt to finish the
      work in progress. This change makes use of the new ability in Go
      1.9 and onwards to cancel a HTTP server gracefully.
      
      The write_timeout duration is used as a grace period to allow all
      in-flight requests to complete. The pattern is taken directly from
      the offical example in the Golang documentation. [1]
      
      Further tuning and testing may be needed for Windows containers which
      have a different set of signals for closing work. This change aims
      to cover the majority use-case for Linux containers.
      
      The HTTP health-check is also invalidated by creating an and
      expression with the existing lock file.
      
      Tested with Kubernetes by deploying a custom watchdog and the
      fprocess of `env`. Log message was observed when scaling down and
      connections stopped being accepted on terminating replica.
      
      Also corrects some typos from previous PR.
      
      [1] https://golang.org/pkg/net/http/#Server.Shutdown
      
      
      
      Signed-off-by: default avatarAlex Ellis (VMware) <alexellis2@gmail.com>
      de2c74fc
  18. Mar 23, 2018
  19. Mar 22, 2018
  20. Mar 20, 2018
  21. Mar 17, 2018
  22. Mar 03, 2018
  23. Feb 27, 2018
  24. Feb 25, 2018
  25. Dec 30, 2017
  26. Dec 28, 2017
  27. Nov 22, 2017
  28. Nov 18, 2017
  29. Nov 09, 2017
Loading