Manually Rotating Control Plane TLS Credentials

Linkerd’s automatic mTLS feature uses a set of TLS credentials to generate TLS certificates for proxies: a trust anchor, and an issuer certificate and private key. The trust anchor has a limited period of validity: 365 days if generated by linkerd install, or a customized value if generated manually.

Thus, for clusters that are expected to outlive this lifetime, you must manually rotate the trust anchor. In this document, we describe how to accomplish this without downtime.

Independent of the trust anchor, the issuer certificate and key pair can also expire (though it is possible to use cert-manager to set up automatic rotation. This document also covers how to rotate the issuer certificate and key pair without downtime.

Prerequisites

These instructions use the following CLI tools:

step to manipulate certificates and keys;

Understanding the current state of your system

Begin by running:

linkerd check --proxy

If your configuration is valid and your credentials are not expiring soon, you should see output similar to:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

However, if you see a message warning you that your trust anchor (“trust root”) or issuer certificates are expiring soon, then you must rotate them.

Note that this document only applies if the trust anchor is currently valid. If your trust anchor has expired, follow the Replacing Expired Certificates Guide instead. (If your issuer certificate has expired but your trust anchor is still valid, continue on with this document.)

For example, if your issuer certificate has expired, you will see a message similar to:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
× issuer cert is within its validity period
issuer certificate is not valid anymore. Expired on 2019-12-19T09:02:01Z
see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-is-time-valid for hints

If your trust anchor has expired, you will see a message similar to:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
× trust roots are within their validity period
Invalid roots:
* 79461543992952791393769540277800684467 identity.linkerd.cluster.local not valid anymore. Expired on 2019-12-19T09:11:30Z
see https://linkerd.io/2/checks/#l5d-identity-roots-are-time-valid  for hints

Rotating the trust anchor

Rotating the trust anchor without downtime is a multi-step process: you must generate a new trust anchor, bundle it with the old one, rotate the issuer certificate and key pair, and finally remove the old trust anchor from the bundle. If you simply need to rotate the issuer certificate and key pair, you can skip directly to Rotating the identity issuer certificate and ignore the trust anchor rotation steps.

Read the current trust anchor certificate from the cluster

To avoid downtime, you need to bundle the existing trust anchor certificate with the newly-generated trust anchor certificate into a certificate bundle: using the bundle allows workloads ultimately signed with either trust anchor to work properly in the mesh. Since certificates are not sensitive information, we can simply pull the existing trust anchor certificate directly from the cluster.

The following command uses kubectl to fetch the Linkerd config from the linkerd-identity-trust-roots ConfigMap and save it in original-trust.crt:

kubectl -n linkerd get cm linkerd-identity-trust-roots -o=jsonpath='{.data.ca-bundle\.crt}' > original-trust.crt

Generate a new trust anchor

After saving the current trust anchor certificate, generate a new trust anchor certificate and private key:

step certificate create root.linkerd.cluster.local ca-new.crt ca-new.key --profile root-ca --no-password --insecure

Note that we use --no-password --insecure to avoid encrypting these files with a passphrase. Store the private key somewhere secure so that it can be used in the future to generate new issuer certificates.

Bundle your original trust anchor with the new one

Next, we need to bundle the trust anchor currently used by Linkerd together with the new anchor. We use step to combine the two certificates into one bundle:

step certificate bundle ca-new.crt original-trust.crt bundle.crt

If desired, you can rm original-trust.crt too.

Deploying the new bundle to Linkerd

At this point you can use the linkerd upgrade command to instruct Linkerd to work with the new trust bundle:

linkerd upgrade --identity-trust-anchors-file=./bundle.crt | kubectl apply -f -

or you can also use the helm upgrade command:

helm upgrade linkerd-control-plane --set-file identityTrustAnchorsPEM=./bundle.crt

Once this is done, you’ll need to restart your meshed workloads so that they use the new trust anchor. For example, doing that for the emojivoto namespace would look like:

kubectl -n emojivoto rollout restart deploy

Now you can run the check command to ensure that everything is ok:

linkerd check --proxy

You might have to wait a few moments until all the pods have been restarted and are configured with the correct trust anchor. Meanwhile you might observe warnings:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2019-12-19T09:51:19Z
    see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
‼ data plane proxies certificate match CA
    Some pods do not have the current trust bundle and must be restarted:
        * emojivoto/emoji-d8d7d9c6b-8qwfx
        * emojivoto/vote-bot-588499c9f6-zpwz6
        * emojivoto/voting-8599548fdc-6v64k
        * emojivoto/web-67c7599f6d-xx98n
        * linkerd/linkerd-sp-validator-75f9d96dc-rch4x
        * linkerd/linkerd-tap-68d8bbf64-mpzgb
        * linkerd/linkerd-web-849f74b7c6-qlhwc
    see https://linkerd.io/2/checks/#l5d-identity-data-plane-proxies-certs-match-ca for hints

When the rollout completes, your check command should stop warning you that pods need to be restarted. It may still warn you, however, that your issuer certificate is about to expire soon:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2019-12-19T09:51:19Z
    see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

At this point, all meshed workloads are ready to accept connections signed by either the old or new trust anchor, but they’re all still using certificates signed by the old trust anchor. To change that, we’ll need to rotate the issuer certificate.

Rotating the identity issuer certificate

To rotate the issuer certificate and key pair, start by generating the new identity issuer certificate and key:

step certificate create identity.linkerd.cluster.local issuer-new.crt issuer-new.key \
--profile intermediate-ca --not-after 8760h --no-password --insecure \
--ca ca-new.crt --ca-key ca-new.key

This new issuer certificate is signed by our new trust anchor, which is why it was critical to install the new trust anchor bundle (as outlined in the previous section). Once the new bundle is installed and running linkerd check shows all green checks and no warnings, you can safely rotate the identity issuer certificate and key by using the upgrade command again:

linkerd upgrade \
    --identity-issuer-certificate-file=./issuer-new.crt \
    --identity-issuer-key-file=./issuer-new.key \
    | kubectl apply -f -

helm upgrade linkerd-control-plane \
  --set-file identity.issuer.tls.crtPEM=./issuer-new.crt \
  --set-file identity.issuer.tls.keyPEM=./issuer-new.key

At this point you can check for the IssuerUpdated Kubernetes event to be certain that Linkerd saw the new issuer certificate:

kubectl get events --field-selector reason=IssuerUpdated -n linkerd

LAST SEEN   TYPE     REASON          OBJECT                        MESSAGE
9s          Normal   IssuerUpdated   deployment/linkerd-identity   Updated identity issuer

Restart the proxy for all injected workloads in your cluster to ensure that their proxies pick up certificates issued by the new issuer:

kubectl -n emojivoto rollout restart deploy

Run the check command to make sure that everything is going as expected:

linkerd check --proxy

You should see output without any certificate expiration warnings (unless an expired trust anchor still needs to be removed):

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

Removing the old trust anchor

Since the old trust anchor is now completely unused, we can now switch Linkerd from the bundle we created for the trust anchor to using only the new trust anchor certificate:

linkerd upgrade  --identity-trust-anchors-file=./ca-new.crt  | kubectl apply -f -

helm upgrade linkerd2 --set-file --set-file identityTrustAnchorsPEM=./ca-new.crt

Note that the ./ca-new.crt file is the same trust anchor you created at the start of this process.

Once again, explicitly restart your meshed workloads:

kubectl -n emojivoto rollout restart deploy
linkerd check --proxy

And, again, the output of the check command should not produce any warnings or errors:

linkerd-identity
----------------
√ certificate config is valid
√ trust roots are using supported crypto algorithm
√ trust roots are within their validity period
√ trust roots are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust root

linkerd-identity-data-plane
---------------------------
√ data plane proxies certificate match CA

Congratulations, you have rotated your trust anchor! 🎉