Manually Rotating Control Plane TLS Credentials
Linkerd’s automatic mTLS feature uses a set of
TLS credentials to generate TLS certificates for proxies: a trust anchor, and
an issuer certificate and private key. The trust anchor has a limited period of
validity: 365 days if generated by linkerd install, or a customized value if
generated manually.
Thus, for clusters that are expected to outlive this lifetime, you must manually rotate the trust anchor. In this document, we describe how to accomplish this without downtime.
Independent of the trust anchor, the issuer certificate and key pair can also
expire (though it is possible to use cert-manager to set up automatic
rotation. This
document also covers how to rotate the issuer certificate and key pair without
downtime.
Prerequisites
These instructions use the step and jq CLI tools.
Understanding the current state of your system
Begin by running:
linkerd check --proxy
If your configuration is valid and your credentials are not expiring soon, you should see output similar to:
linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root
linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA
However, if you see a message warning you that your trust anchor (“trust root”) or issuer certificates are expiring soon, then you must rotate them.
Note that this document only applies if the trust root and issuer certificate are currently valid. If your trust anchor or issuer certificate have expired, please follow the Replacing Expired Certificates Guide instead.
For example, if your issuer certificate has expired, you will see a message similar to:
linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
× issuer cert is within its validity period
issuer certificate is not valid anymore. Expired on 2019-12-19T09:02:01Z
see https://linkerd.io/checks/#l5d-identity-issuer-cert-is-time-valid for hints
If your trust anchor has expired, you will see a message similar to:
linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
× trust roots are within their validity period
Invalid roots:
* 79461543992952791393769540277800684467 identity.linkerd.cluster.local not valid anymore. Expired on 2019-12-19T09:11:30Z
see https://linkerd.io/checks/#l5d-identity-roots-are-time-valid for hints
Rotating the trust anchor
Rotating the trust anchor without downtime is a multi-step process: you must generate a new trust anchor, bundle it with the old one, rotate the issuer certificate and key pair, and finally remove the old trust anchor from the bundle. If you simply need to rotate the issuer certificate and key pair, you can skip directly to Rotating the identity issuer certificate and ignore the trust anchor rotation steps.
Generate a new trust anchor
First, generate a new trust anchor certificate and private key:
step certificate create root.linkerd.cluster.local ca-new.crt ca-new.key --profile root-ca --no-password --insecure
Note that we use --no-password --insecure to avoid encrypting these files
with a passphrase. Store the private key somewhere secure so that it can be
used in the future to generate new issuer
certificates.
Bundle your original trust anchor with the new one
Next, we need to bundle the trust anchor currently used by Linkerd together with
the new anchor. The following command uses kubectl to fetch the Linkerd config,
jq/yq to extract the current trust anchor,
and step to combine it with the newly generated trust anchor:
# For Linkerd < 2.9.0:
kubectl -n linkerd get cm linkerd-config -o=jsonpath='{.data.global}' \
| jq -r .identityContext.trustAnchorsPem > original-trust.crt
# For Linkerd >= 2.9.0:
kubectl -n linkerd get cm linkerd-config -o=jsonpath='{.data.values}' \
| yq e .global.identityTrustAnchorsPEM - > original-trust.crt
step certificate bundle ca-new.crt original-trust.crt bundle.crt
rm original-trust.crt
Deploying the new bundle to Linkerd
At this point you can use the linkerd upgrade command to instruct Linkerd to
work with the new trust bundle:
linkerd upgrade --identity-trust-anchors-file=./bundle.crt | kubectl apply -f -
or you can also use the helm upgrade command:
helm upgrade linkerd2 --set-file global.identityTrustAnchorsPEM=./bundle.crt
This will restart the proxies in the Linkerd control plane, and they will be reconfigured with the new trust anchor.
Finally, you must restart the proxy for all injected workloads in your cluster.
For example, doing that for the emojivoto namespace would look like:
kubectl -n emojivoto rollout restart deploy
Now you can run the check command to ensure that everything is ok:
linkerd check --proxy
You might have to wait a few moments until all the pods have been restarted and are configured with the correct trust anchor. Meanwhile you might observe warnings:
linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2019-12-19T09:51:19Z
see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust root
linkerd-identity-data-plane
---------------------------
‼ data plane proxies certificate match CA
Some pods do not have the current trust bundle and must be restarted:
* emojivoto/emoji-d8d7d9c6b-8qwfx
* emojivoto/vote-bot-588499c9f6-zpwz6
* emojivoto/voting-8599548fdc-6v64k
* emojivoto/web-67c7599f6d-xx98n
* linkerd/linkerd-sp-validator-75f9d96dc-rch4x
* linkerd/linkerd-tap-68d8bbf64-mpzgb
* linkerd/linkerd-web-849f74b7c6-qlhwc
see https://linkerd.io/checks/#l5d-identity-data-plane-proxies-certs-match-ca for hints
When the rollout completes, your check command should stop warning you that
pods need to be restarted. It may still warn you, however, that your issuer
certificate is about to expire soon:
linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2019-12-19T09:51:19Z
see https://linkerd.io/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust root
linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA
Rotating the identity issuer certificate
To rotate the issuer certificate and key pair, first generate a new pair:
step certificate create identity.linkerd.cluster.local issuer-new.crt issuer-new.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca-new.crt --ca-key ca-new.key
Provided that the trust anchor has not expired and that, if recently rotated,
all proxies have been updated to include a working trust anchor (as outlined in
the previous section) it is now safe to rotate the identity issuer certificate
by using the upgrade command again:
linkerd upgrade --identity-issuer-certificate-file=./issuer-new.crt --identity-issuer-key-file=./issuer-new.key | kubectl apply -f -
or
exp=$(cat ca-new.crt | openssl x509 -noout -dates | grep "notAfter" | sed -e 's/notAfter=\(.*\)$/"\1"/' | TZ='GMT' xargs -I{} date -d {} +"%Y-%m-%dT%H:%M:%SZ")
helm upgrade linkerd2
--set-file identity.issuer.tls.crtPEM=./issuer-new.crt
--set-file identity.issuer.tls.keyPEM=./issuer-new.key
--set identity.issuer.crtExpiry=$exp
At this point Linkerd’s identity control plane service should detect the
change of the secret and automatically update its issuer certificates.
To ensure this has happened, you can check for the specific Kubernetes event:
kubectl get events --field-selector reason=IssuerUpdated -n linkerd
LAST SEEN TYPE REASON OBJECT MESSAGE
9s Normal IssuerUpdated deployment/linkerd-identity Updated identity issuer
Restart the proxy for all injected workloads in your cluster to ensure that their proxies pick up certificates issued by the new issuer:
kubectl -n emojivoto rollout restart deploy
Run the check command to make sure that everything is going as expected:
linkerd check --proxy
You should see output without any certificate expiration warnings (unless an expired trust anchor still needs to be removed):
linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root
linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA
Removing the old trust anchor
We can now remove the old trust anchor from the trust bundle we created earlier.
The upgrade command can do that for the Linkerd components:
linkerd upgrade --identity-trust-anchors-file=./ca-new.crt | kubectl apply -f -
or
helm upgrade linkerd2 --set-file --set-file global.identityTrustAnchorsPEM=./ca-new.crt
Note that the ./ca-new.crt file is the same trust anchor you created at the start
of this process. Additionally, you can use the rollout restart command to
bring the configuration of your other injected resources up to date:
kubectl -n emojivoto rollout restart deploy
linkerd check --proxy
Finally the output of the check command should not produce any warnings or
errors:
linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root
linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA
Congratulations, you have rotated your trust anchor! 🎉