Graceful Pod Shutdown
When Kubernetes begins to terminate a pod, it starts by sending all containers in that pod a TERM signal. When the Linkerd proxy sidecar receives this signal, it will immediately begin a graceful shutdown where it refuses all new requests and allows existing requests to complete before shutting down.
This means that if the pod’s main container attempts to make any new network calls after the proxy has received the TERM signal, those network calls will fail. This also has implications for clients of the terminating pod and for job resources.
Slow Updating Clients
Before Kubernetes terminates a pod, it first removes that pod from the endpoints resource of any services that pod is a member of. This means that clients of that service should stop sending traffic to the pod before it is terminated. However, certain clients can be slow to receive the endpoints update and may attempt to send requests to the terminating pod after that pod’s proxy has already received the TERM signal and begun graceful shutdown. Those requests will fail.
To mitigate this, use the
--wait-before-exit-seconds flag with
linkerd inject to delay the Linkerd proxy’s handling of the TERM signal for
a given number of seconds using a
preStop hook. This delay gives slow clients
additional time to receive the endpoints update before beginning graceful
shutdown. To achieve max benefit from the option, the main container should have
preStop hook with the sleep command inside which has a smaller period
than is set for the proxy sidecar. And none of them must be bigger than
terminationGracePeriodSeconds configured for the entire pod.
# application container lifecycle: preStop: exec: command: - /bin/bash - -c - sleep 20 # for entire pod terminationGracePeriodSeconds: 160
Graceful shutdown of Job and Cronjob Resources
Pods which are part of Job or Cronjob resources will run until all of the containers in the pod complete. However, the Linkerd proxy container runs continuously until it receives a TERM signal. Since Kubernetes does not give the proxy a means to know when the Cronjob has completed, by default, Job and Cronjob pods which have been meshed will continue to run even once the main container has completed.
To address this, you can issue a POST to the
/shutdown endpoint on the proxy
once the application completes (e.g. via
curl -X POST
http://localhost:4191/shutdown). This will terminate the proxy gracefully and
allow the Job or Cronjob to complete. These shutdown requests must come on the
loopback interface, i.e. from within the same Kubernetes pod.
One convenient way to call this endpoint is to wrap your application with the
linkerd-await utility. An
application that is called this way (e.g. via
linkerd-await -S $MYAPP) will
automatically call the proxy’s
/shutdown endpoint when it completes.
In the future, Kubernetes will hopefully support more container lifecycle hooks that will allow Linkerd to handle these situations automatically.