-
- Downloads
Alter graceful shutdown sequence
- the shutdown sequence meant that the kubelet was still passing
work to the watchdog after the HTTP socket was closed. This change
means that the kubelet has a chance to run its check before we
finally stop accepting new connections. It will require some
basic co-ordination between the kubelet's checking period and the
"write_timeout" value in the container.
Tested with Kubernetes on GKE - before the change some Pods were
giving a connection refused error due to them being not detected
as unhealthy. Now I receive 0% error rate even with 20 qps.
Issue was shown by scaling to 20 replicas, starting a test with
hey and then scaling to 1 replica while tailing the logs from the
gateway. Before I saw some 502, now I see just 200s.
Signed-off-by:
Alex Ellis (VMware) <alexellis2@gmail.com>
Showing
- watchdog/README.md 10 additions, 0 deletionswatchdog/README.md
- watchdog/build.sh 3 additions, 1 deletionwatchdog/build.sh
- watchdog/handler.go 6 additions, 5 deletionswatchdog/handler.go
- watchdog/main.go 49 additions, 20 deletionswatchdog/main.go
- watchdog/requesthandler_test.go 1 addition, 1 deletionwatchdog/requesthandler_test.go
Loading
Please register or sign in to comment