Automatically Rotating Control Plane TLS Credentials

Linkerd’s automatic mTLS, like any TLS implementation, relies on properly-configured certificates as the basis of identity within the mesh. Each meshed workload has its own workload certificate generated by Linkerd itself based on a trust anchor, which can be shared across clusters, and an identity issuer certificate, which is specific to the cluster.

While Linkerd automatically rotates the workload certificates, it cannot automatically rotate the identity issuer certificate or the trust anchor. Linkerd’s out-of-the-box installations generate static self-signed certificates with a validity of one year but require manual rotation by the user to prevent expiry. While this setup is convenient for quick start testing, it’s not advisable nor recommended for production environments.

Automating Certificate Management with cert-manager and trust-manager

cert-manager and trust-manager are popular CNCF tools that automate certificate management for Kubernetes installations. They can work together with Linkerd to automate rotating the identity issuer certificate and partly automate rotating the trust anchor.

cert-manager is extremely flexible, and much of its configuration depends on the specific policies of the organization running it. Rather than attempt to provide a comprehensive guide to cert-manager, this document will focus on a very simple setup:

• cert-manager will create a self-signed trust anchor.

  Since the trust anchor is a self-signed certificate, its private key is stored in the cluster (specifically, in a Kubernetes Secret in the cert-manager namespace). It would be more secure to keep the trust anchor’s private key off the cluster entirely, since Linkerd never needs access to it; we’ll discuss that a bit more in the trust anchor setup section.

• cert-manager will use the trust anchor to create Linkerd’s identity issuer certificate.

  The identity issuer certificate’s private key is also stored in the cluster (in a Secret in the linkerd namespace). Linkerd does need access to this private key; this is the only way to manage the identity issuer.

• Finally, trust-manager will create a trust bundle that Linkerd will use to verify the authenticity of certificates issued by cert-manager.

  The trust bundle doesn’t contain any private keys at all. It will be stored in a ConfigMap in the linkerd namespace.

Once the certificates are created, cert-manager will automatically rotate the identity issuer certificate as necessary. Rotating the trust anchor is a bit more complex: though cert-manager can do the heavy lifting for you, the rotation will still involve manual intervention, as explained below.

Setup Overview

The process we will follow is straightforward even though it has several steps:

1. Create the linkerd namespace in which we need our Linkerd certificates to live
2. Install cert-manager and trust-manager on your cluster
3. Configure cert-manager to create the trust anchor
4. Configure cert-manager to create the identity issuer certificate
5. Configure cert-manager to create a trust bundle for Linkerd to use
6. Install Linkerd using certificates created by cert-manager
7. Check everything that cert-manager did!
8. Rotating the identity issuer
9. Rotating the trust anchor

1. Create the linkerd namespace

This may seem a bit odd, since we haven’t installed Linkerd yet! However, this is an important first step: Linkerd expects its certificates to be in the linkerd namespace, and when working with cert-manager, Linkerd needs the certificates to already be present when it is installed. So we’ll create the namespace now:

kubectl create namespace linkerd

2. Install cert-manager and trust-manager

Next up, install cert-manager (we’ll use Helm for this, but you can check out the cert-manager installation guide for more options):

helm repo add jetstack https://charts.jetstack.io --force-update

helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true

kubectl rollout status -n cert-manager deploy

We strongly recommend installing cert-manager in the cert-manager namespace.

Once cert-manager is installed, install trust-manager (again, we’ll use Helm for this, but there are more options in the trust-manager installation guide). We’ll install trust-manager in the cert-manager namespace as well, and we’ll also configure trust-manager to use the cert-manager namespace as its trust namespace. The trust namespace is the only namespace from which trust-manager is allowed to read Secrets: since we want trust-manager to look at Secrets for the certificates that cert-manager is creating, the cert-manager namespace is the one we want.

helm install \
  trust-manager jetstack/trust-manager \
  --namespace cert-manager \
  --set app.trust.namespace=cert-manager \
  --wait

Finally, we’ll need to update cert-manager’s RBAC permissions. By default cert-manager will only create certificate secrets in the namespace where it is installed. Linkerd, however, requires its identity issuer to be created in the linkerd namespace. To allow this, we create a ServiceAccount for cert-manager in the linkerd namespace with the required permissions.

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cert-manager
  namespace: linkerd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cert-manager-secret-creator
  namespace: linkerd
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cert-manager-secret-creator-binding
  namespace: linkerd
subjects:
  - kind: ServiceAccount
    name: cert-manager
    namespace: linkerd
roleRef:
  kind: Role
  name: cert-manager-secret-creator
  apiGroup: rbac.authorization.k8s.io
EOF

3. Configure cert-manager to create the trust anchor

As described in Buoyant’s cert-manager concepts primer, cert-manager uses issuers to create certificates. Any certificate created and managed by cert-manager must be configured with a Certificate resource and must be linked to an issuer. Any issuer that cert-manager uses must be configured with an Issuer or ClusterIssuer resource.

The main difference between an Issuer and a ClusterIssuer is that Issuers can only be used by Certificates in the same namespace as the Issuer, while ClusterIssuers can cross namespaces. For the trust anchor, we’ll use an Issuer in the cert-manager namespace. This will be a self-signed issuer, meaning that will simply generate self-signed certificates with random keys – this is the simplest kind of issuer.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  # This is the name of the Issuer resource; it's the way
  # Certificate resources can find this issuer.
  name: linkerd-trust-root-issuer
  namespace: cert-manager
spec:
  selfSigned: {}
EOF

Next, we’ll create a cert-manager Certificate resource which uses the previously-created Issuer.

kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  # This is the name of the Certificate resource, but the Secret
  # we save the certificate into can be different.
  name: linkerd-trust-anchor
  namespace: cert-manager
spec:
  # This tells cert-manager which issuer to use for this Certificate:
  # in this case, the Issuer named linkerd-trust-root-issuer.
  issuerRef:
    kind: Issuer
    name: linkerd-trust-root-issuer

  # The issued certificate will be saved in this Secret
  secretName: linkerd-trust-anchor

  # These are details about the certificate to be issued: check
  # out the cert-manager docs for more, but realize that setting
  # the private key's rotationPolicy to Always is _very_ important,
  # and that for Linkerd you _must_ set isCA to true!
  isCA: true
  commonName: root.linkerd.cluster.local
  # This is a one-year duration, rotating two months before expiry.
  # Feel free to reduce this, but remember that there is a manual
  # process for rotating the trust anchor!
  duration: 8760h0m0s
  renewBefore: 7320h0m0s
  privateKey:
    rotationPolicy: Always
    algorithm: ECDSA
EOF

Note that this Certificate lives in the cert-manager namespace with the linkerd-trust-root-issuer Issuer. cert-manager will write the newly-created certificate into a Secret with the name given by the secretName field, in the same namespace as the Certificate. While Linkerd needs its trust bundle to be in the linkerd namespace, Linkerd does not need access to the private key of the trust anchor, so it’s better to keep that Secret in the cert-manager namespace where Linkerd can be prevented from accessing it.

At this point, you should see a Secret named linkerd-trust-anchor in the cert-manager namespace:

kubectl get secret -n cert-manager linkerd-trust-anchor

4. Configure cert-manager to create the identity issuer certificate

We now need to configure cert-manager to create the Linkerd identity issuer certificate, which requires creating another issuer. For this, we’ll use a CA-type ClusterIssuer, since we’re going to want cert-manager to use the trust anchor certificate it just created to issue a second certificate.

This needs to be a ClusterIssuer because Linkerd does need access to the private key of the identity issuer certificate, so its Certificate needs to be in the linkerd namespace. Using a ClusterIssuer is the simplest way to cross namespace boundaries here.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  # This is the name of the Issuer resource; it's the way
  # Certificate resources can find this issuer.
  name: linkerd-identity-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: linkerd-trust-anchor
EOF

Next we’ll create a Certificate resource which uses the linkerd-identity-issuer ClusterIssuer to create the Linkerd identity issuer certificate. Linkerd will use this certificate to issue workload certificates to all the Linkerd proxies in the system, so although this Certificate will reference the ClusterIssuer we just created in the cert-manager namespace, the Certificate itself must be in the linkerd namespace.

kubectl apply -f - <<EOF
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  # This is the name of the Certificate resource, but the Secret
  # we save the certificate into can be different.
  name: linkerd-identity-issuer
  namespace: linkerd
spec:
  # This tells cert-manager which issuer to use for this Certificate:
  # in this case, the ClusterIssuer named linkerd-identity-issuer.
  issuerRef:
    name: linkerd-identity-issuer
    kind: ClusterIssuer

  # The issued certificate will be saved in this Secret.
  secretName: linkerd-identity-issuer

  # These are details about the certificate to be issued: check
  # out the cert-manager docs for more, but realize that setting
  # the private key's rotationPolicy to Always is _very_ important,
  # and that for Linkerd you _must_ set isCA to true!
  isCA: true
  commonName: identity.linkerd.cluster.local
  # This is a two-day duration, rotating slightly over a day before
  # expiry. Feel free to set this as you like.
  duration: 48h0m0s
  renewBefore: 25h0m0s
  privateKey:
    rotationPolicy: Always
    algorithm: ECDSA
EOF

At this point, you should see a Secret named linkerd-identity-issuer in the linkerd namespace:

kubectl get secret -n linkerd linkerd-identity-issuer

5. Configure cert-manager to create a trust bundle for Linkerd to use

Almost done! We only need one more thing: the trust bundle will lets Linkerd know which trust anchors to accept. We’ll use trust-manager for this, but there’s a catch: when rotating the trust anchor, both the control plane and the data plane (the proxies) need to be restarted. Since that can’t happen instaneously, we need to have both the old trust anchor and the new trust anchor in the trust bundle until all the restarts have completed.

trust-manager can do this, but it needs a specific source for each certificate in the bundle. So we’ll start by copying the trust anchor from the linkerd-trust-anchor Secret into a second Secret, linkerd-previous-anchor, and then we’ll configure trust-manager to use both Secrets as sources for the trust bundle.

kubectl get secret -n cert-manager linkerd-trust-anchor -o yaml \
        | sed -e s/linkerd-trust-anchor/linkerd-previous-anchor/ \
        | egrep -v '^  *(resourceVersion|uid)' \
        | kubectl apply -f -

This way, when cert-manager rotates the trust anchor and updates the linkerd-trust-anchor Secret, trust-manager will take the new anchor from the linkerd-trust-anchor Secret and the previous anchor from the linkerd-previous-anchor Secret, bundle them together, and save the bundle in a ConfigMap. After everything is restarted, we’ll copy the new trust anchor across to the linkerd-previous-anchor Secret, and since both Secrets are identical, the ConfigMap will only contain a single anchor in a bundle.

Once that’s done, we can create the Bundle resource to tell trust-manager how to build the trust bundle.

kubectl apply -f - <<EOF
---
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  # This is the name of the Bundle and _also_ the name of the
  # ConfigMap in which we'll write the trust bundle.
  name: linkerd-identity-trust-roots
  namespace: linkerd
spec:
  # This tells trust-manager where to find the public keys to copy into
  # the trust bundle.
  sources:
    # This is the Secret that cert-manager will update when it rotates
    # the trust anchor.
    - secret:
        name: "linkerd-trust-anchor"
        key: "tls.crt"

    # This is the Secret that we will use to hold the previous trust
    # anchor; we'll manually update this Secret after we're finished
    # restarting things.
    - secret:
        name: "linkerd-previous-anchor"
        key: "tls.crt"

  # This tells trust-manager the key to use when writing the trust
  # bundle into the ConfigMap. The target stanza doesn't have a way
  # to specify the name of the namespace, but thankfully Linkerd puts
  # a unique label on the control plane's namespace.
  target:
    configMap:
      key: "ca-bundle.crt"
    namespaceSelector:
      matchLabels:
        linkerd.io/is-control-plane: "true"
EOF

You won’t actually see the linkerd-identity-trust-roots ConfigMap in the linkerd namespace yet, because the namespace won’t have the label that trust-manager is looking for until we install Linkerd! So let’s go ahead and get Linkerd installed.

6. Install Linkerd using certificates created by cert-manager

To have Linkerd use the certificates created by cert-manager, you need to add the following to your values.yaml file or pass them in as flags at runtime.

Installing with Helm (recommended)

For installing with Helm, first install the linkerd-crds chart:

helm install linkerd-crds \
     -n linkerd --create-namespace \
     linkerd/linkerd-crds

Then install the linkerd-control-plane chart:

helm install linkerd-control-plane -n linkerd \
  --set identity.externalCA=true \
  --set identity.issuer.scheme=kubernetes.io/tls \
  linkerd/linkerd-control-plane

We’ll also need to label the linkerd namespace with the label that trust-manager will be looking for:

kubectl label namespace linkerd linkerd.io/is-control-plane=true

Voila! We have set up automatic rotation of Linkerd’s control plane TLS credentials.

Installing with the CLI

First, install the CRDs:

linkerd install --crds | kubectl apply -f -

Then install the control plane:

linkerd install \
  --set identity.externalCA=true \
  --set identity.issuer.scheme=kubernetes.io/tls \
  | kubectl apply -f -

Voila! We have set up automatic rotation of Linkerd’s control plane TLS credentials.

7. Check everything that cert-manager did!

You can skip this step if you’re in a hurry, but it’s a good idea to know how to check Linkerd’s trust setup! There are a few ways to do this, but one of the easiest uses the step CLI to inspect the actual certificates stored in the various Kubernetes objects that cert-manager has just set up for us.

First up, let’s look at the actual trust anchor secret. This is the linkerd-trust-anchor Secret in the cert-manager namespace; kubectl describe will show that it has keys of tls.key, tls.crt, and ca.crt:

kubectl describe secret -n cert-manager linkerd-trust-anchor

We won’t look at tls.key - that’s the private key! - but tls.crt is its base64-encoded public key, and we can inspect that:

kubectl get secret -n cert-manager linkerd-trust-anchor \
        -o jsonpath='{ .data.tls\.crt }' \
    | base64 -d | step certificate inspect -

There’s a lot of information in there: things worth checking over include the Issuer, the Subject, the Validity timestamps, etc. But we can use a shell function to easier to see the chain of trust where one certificate signs another:

inspect_cert () {
  sub_selector='\(.extensions.subject_key_id | .[0:16])... \(.subject_dn)'
  iss_selector='\(.extensions.authority_key_id // "................" | .[0:16])... \(.issuer_dn)'

  step certificate inspect --format json "$1" \
    | jq -r "\"Issuer:  $iss_selector\",\"Subject: $sub_selector\""
}

kubectl get secret -n cert-manager linkerd-trust-anchor \
        -o jsonpath='{ .data.tls\.crt }' \
    | base64 -d | inspect_cert

We should see something like this:

Issuer:  ................... CN=root.linkerd.cluster.local
Subject: 5c455d3e9bd77e91... CN=root.linkerd.cluster.local

where the Subject’s key fingerprint - the hex number on the second line - will of course be different for your certificate.

Since our setup uses a self-signed certificate (as expected!), we won’t see an issuer fingerprint, and the issuer and subject names will be the same, and the ca.crt key should have exactly the same information as tls.crt:

kubectl get secret -n cert-manager linkerd-trust-anchor \
        -o jsonpath='{ .data.ca\.crt }' \
    | base64 -d | inspect_cert

This output should look exactly the same as the tls.crt output.

Since we copied the trust anchor to the linkerd-previous-anchor Secret, too, we should see exactly the same information there as we do in the linkerd-trust-anchor Secret:

kubectl get secret -n cert-manager linkerd-previous-anchor \
        -o jsonpath='{ .data.tls\.crt }' \
    | base64 -d | inspect_cert

kubectl get secret -n cert-manager linkerd-previous-anchor \
        -o jsonpath='{ .data.ca\.crt }' \
    | base64 -d | inspect_cert

Next, we can check the identity issuer certificate. This is the linkerd-identity-issuer Secret in the linkerd namespace, and it will appear to have exactly the same structure as the trust anchor secret:

kubectl describe secret -n linkerd linkerd-identity-issuer

The ca.crt key should be exactly the same as what we just saw from the trust anchor, at this point:

kubectl get secret -n linkerd linkerd-identity-issuer \
        -o jsonpath='{ .data.ca\.crt }' \
    | base64 -d | inspect_cert

The tls.crt key should be the public key of the identity issuer certificate, so its Issuer line should show the trust anchor’s fingerprint, and its Subject line should be different.

kubectl get secret -n linkerd linkerd-identity-issuer \
        -o jsonpath='{ .data.tls\.crt }' \
    | base64 -d | inspect_cert

Here, we should see something like

Issuer:  5c455d3e9bd77e91... CN=root.linkerd.cluster.local
Subject: 56bfe071553c16ad... CN=identity.linkerd.cluster.local

where the Issuer line should be exactly the same as the Subject line from the previous inspections (and, again, your hex values will be different from the ones shown above).

Finally, we can check the trust bundle. This is the linkerd-identity-trust-roots ConfigMap in the linkerd namespace, and it should have a key named ca-bundle.crt. This key should contain our trust bundle, which is a set of public keys – at the moment, there should just be one key, the public key of our current trust anchor. This is not base64-encoded, so we can start by dumping it directly:

kubectl get configmap -n linkerd linkerd-identity-trust-roots \
        -o jsonpath='{ .data.ca-bundle\.crt }'

This should be a single PEM CERTIFICATE block:

-----BEGIN CERTIFICATE-----
...lots of random-looking stuff here...
-----END CERTIFICATE-----

and if we pipe it to inspect_cert, we should once again see the trust anchor’s information.

kubectl get configmap -n linkerd linkerd-identity-trust-roots \
        -o jsonpath='{ .data.ca-bundle\.crt }' \
    | inspect_cert

8. Rotating the identity issuer

Rotating the identity issuer is basically a non-event: cert-manager can handle rotating the identity issuer completely on its own. When it does so, it will update the linkerd-identity-issuer Secret in the linkerd namespace, at which point every Linkerd proxy will automatically notice this change and start using the new certificate for issuing workload certificates. Since the trust anchor hasn’t changed, nothing further is needed and everything will continue smoothly.

You can monitor identity issuer rotation by checking the IssuerUpdated events emitted by Linkerd:

kubectl get events --field-selector reason=IssuerUpdated -n linkerd

9. Rotating the trust anchor

Rotating the trust anchor is a bit different, because (as mentioned before) rotating the trust anchor mean that you have to restart both the Linkerd control plane and all the proxies while managing the trust bundle. In practice, this requires manual intervention, because while cert-manager can handle the hard work of actually rotating the trust anchor, it can’t trigger the needed restarts.

This means that the simplest way to handle trust anchor rotation is to *trigger the rotation manually whenever it’s convenient for you so that you can manage the trust bundle and restarts while letting cert-manager manage the trust anchor certificate.

The process of actually doing this is straightforward, but again, there are several steps:

1. Trigger trust anchor rotation
2. Trigger identity issuer rotation
3. Restart the control plane
4. Restart the data plane
5. Remove the old anchor from the trust bundle

1. Triggering trust anchor rotation

Start by triggering cert-manager to rotate the trust anchor. The easiest way to do this is with cert-manager’s cmctl CLI:

cmctl renew -n cert-manager linkerd-trust-anchor

This will cause cert-manager to rotate the trust anchor certificate, which will update the linkerd-trust-anchor Secret, which will trigger trust-manager to update the linkerd-identity-trust-roots ConfigMap. You can (and should!) check both of these things with kubectl – the linkerd-trust-anchor and linkerd-previous-anchor should no longer show the same Subject keys:

kubectl get secret -n cert-manager linkerd-trust-anchor \
        -o jsonpath='{ .data.tls\.crt }' \
    | base64 -d | inspect_cert

kubectl get secret -n cert-manager linkerd-previous-anchor \
        -o jsonpath='{ .data.tls\.crt }' \
    | base64 -d | inspect_cert

Additionally, the linkerd-identity-trust-roots ConfigMap should now contain the Subject keys from both Secrets. (We can’t use inspect_cert for this since there are two keys.)

kubectl get configmap -n linkerd linkerd-identity-trust-roots \
        -o jsonpath='{ .data.ca-bundle\.crt }' \
        | step certificate inspect --bundle --format json \
        | jq -r ".[] | \"Subject: \(.extensions.subject_key_id | .[0:16])... \(.subject_dn)\""

Note that mTLS communication is still working fine at this point. All the proxies in the data plane are still using the old identity issuer, signed by the old trust anchor, and that trust anchor is still present in linkerd-identity-trust-roots.

2. Triggering identity issuer rotation

Everything is still using the old identity issuer because, strangely, manually triggering cert-manager to rotate the trust anchor does not automatically rotate the identity issuer. You can verify this by doublechecking the identity issuer directly:

kubectl get secret -n linkerd linkerd-identity-issuer \
        -o jsonpath='{ .data.tls\.crt }' \
    | base64 -d | inspect_cert

You’ll see that its issuer is still the old trust anchor. Our next step, therefore, is to trigger cert-manager to rotate the identity issuer.

cmctl renew -n linkerd linkerd-identity-issuer

If you re-check the identity issuer now, you’ll see that it’s signed by the new trust anchor:

kubectl get secret -n linkerd linkerd-identity-issuer \
        -o jsonpath='{ .data.tls\.crt }' \
    | base64 -d | inspect_cert

3. Restart the control plane

Restart the control plane with kubectl rollout restart:

kubectl rollout restart -n linkerd deploy
kubectl rollout status -n linkerd deploy

This will cause the control plane to pick up the new trust anchor and identity issuer.

4. Restart the data plane

At this point, you’ll need to restart your workloads as well, to force the proxies to switch to the new identity issuer. The exact mechanism to do this when depend on your workloads, but it’s often just kubectl rollout restart for each of your application namespaces.

5. Remove the old anchor from the trust bundle

One last step: once everything is restarted, you’ll need to remove the old trust anchor from the trust bundle. To do this, just copy the linkerd-trust-anchor Secret to the linkerd-previous-anchor Secret; that will trigger trust-manager to update the trust bundle ConfigMap.

kubectl get secret -n cert-manager linkerd-trust-anchor -o yaml \
        | sed -e s/linkerd-trust-anchor/linkerd-previous-anchor/ \
        | egrep -v '^  *(resourceVersion|uid)' \
        | kubectl apply -f -

You can doublecheck this with kubectl again:

kubectl get configmap -n linkerd linkerd-identity-trust-roots \
        -o jsonpath='{ .data.ca-bundle\.crt }' \
        | step certificate inspect --format json \
        | jq -r "\"Subject: \(.extensions.subject_key_id | .[0:16])... \(.subject_dn)\""

and you should only see the single ID of the current trust anchor.

At this point rotation is complete: everything is using the new trust anchor, and the old trust anchor is no longer trusted.

See also

• Automatically Rotating Webhook TLS Credentials
• Manually rotating Linkerd’s trust anchor credentials