The Gotchas of Zero-Downtime Traffic on Kubernetes /w Leigh Capili

Denver DevOps Meetup May 8, 2019

Shaping Traffic

• Load balancers are layer 4 or 7
• They accept traffic and direct the requests
• Connection draining is when you reduce and then turn off the ability to accept connections
  • SIGHUP or SIGTERM are some commands that help processes exit gracefully. These are used to initiate connection draining

Shaping Traffic in Kubernetes

• In k8s, pods can report on readiness. Both if the process is up and running, and (if configured) the pod is returning an HTTP status code
• How pods are shutdown:
  • kube-apiserver receives delete
  • pod marked as terminating (visible from the console) and unready (secretly)
  • prestop hooks run
  • PID 1 receive SIGTERM
  • Termination grace period of 30 seconds
  • If the container hasn't responded to SIGTERM, PID 1 of all containers receive SIGKILL

Gotchas

1. Signals.
   • Solution: Dockerfiles should call command as an array.

• CMD nginx vs.
• `ENTRYPOINT ["nginx]

2. Stopsignal

• ?

3. Readiness and Liveness Probes

• Liveness is used to check if process is OK
• Readiness is used to check if the pod should receive traffic
• Solution: be intentional with timeouts and periods.
• Liveness and readiness probe values should never be the same

4. Prestop lifecycle hook

• Important because the defintion of graceful shutdown for most program is dissonant with K8s expectations
  • stop receiving connections vs. start draining connections
• Make your own graceful shutdown by writing a script that lives in your nginx container that calls kill and then sleeps. '