Using GitOps with Linkerd with Argo CD
GitOps is an approach to automate the management and delivery of your Kubernetes infrastructure and applications using Git as a single source of truth. It usually utilizes some software agents to detect and reconcile any divergence between version-controlled artifacts in Git with what’s running in a cluster.
This guide will show you how to set up Argo CD to manage the installation and upgrade of Linkerd using a GitOps workflow.
Linkerd Production Tip
Specifically, this guide provides instructions on how to securely generate and manage Linkerd’s mTLS private keys and certificates using Sealed Secrets and cert-manager. It will also show you how to integrate the auto proxy injection feature into your workflow. Finally, this guide conclude with steps to upgrade Linkerd to a newer version following a GitOps workflow.
The software and tools used in this guide are selected for demonstration purposes only. Feel free to choose others that are most suited for your requirements.
You will need to clone this example repository to your local machine and replicate it in your Kubernetes cluster following the steps defined in the next section.
This guide uses the step cli to create certificates used by the Linkerd clusters to enforce mTLS, so make sure you have installed step for your environment.
Set up the repositories
Clone the example repository to your local machine:
git clone https://github.com/linkerd/linkerd-examples.git
This repository will be used to demonstrate Git operations like add
, commit
and push
later in this guide.
Add a new remote endpoint to the repository to point to the in-cluster Git server, which will be set up in the next section:
cd linkerd-examples
git remote add git-server git://localhost/linkerd-examples.git
Note
Deploy the Git server to the scm
namespace in your cluster:
kubectl apply -f gitops/resources/git-server.yaml
Later in this guide, Argo CD will be configured to watch the repositories hosted by this Git server.
Note
git
protocol, with unauthenticated access to the Git data. This setup is not
recommended for production use.Confirm that the Git server is healthy:
kubectl -n scm rollout status deploy/git-server
Clone the example repository to your in-cluster Git server:
git_server=`kubectl -n scm get po -l app=git-server -oname | awk -F/ '{ print $2 }'`
kubectl -n scm exec "${git_server}" -- \
git clone --bare https://github.com/linkerd/linkerd-examples.git
Confirm that the remote repository is successfully cloned:
kubectl -n scm exec "${git_server}" -- ls -al /git/linkerd-examples.git
Confirm that you can push from the local repository to the remote repository via port-forwarding:
kubectl -n scm port-forward "${git_server}" 9418 &
git push git-server master
Install the Argo CD CLI
Before proceeding, install the Argo CD CLI in your local machine by following the instructions relevant to your OS.
Deploy Argo CD
Install Argo CD:
kubectl create ns argocd
kubectl -n argocd apply -f \
https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
Confirm that all the pods are ready:
for deploy in "dex-server" "redis" "repo-server" "server"; \
do kubectl -n argocd rollout status deploy/argocd-${deploy}; \
done
kubectl -n argocd rollout status statefulset/argocd-application-controller
Use port-forward to access the Argo CD dashboard:
kubectl -n argocd port-forward svc/argocd-server 8080:443 \
> /dev/null 2>&1 &
The Argo CD dashboard is now accessible at
https://localhost:8080, using the default admin
username and
password.
Authenticate the Argo CD CLI:
password=`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`
argocd login 127.0.0.1:8080 \
--username=admin \
--password="${password}" \
--insecure
Configure project access and permissions
Set up the demo
project to group our
applications:
kubectl apply -f gitops/project.yaml
This project defines the list of permitted resource kinds and target clusters that our applications can work with.
Confirm that the project is deployed correctly:
argocd proj get demo
On the dashboard:
Deploy the applications
Deploy the main
application which serves as the “parent” for all the other
applications:
kubectl apply -f gitops/main.yaml
Note
Confirm that the main
application is deployed successfully:
argocd app get main
Sync the main
application:
argocd app sync main
Notice that only the main
application is synchronized.
Next, we will synchronize the remaining applications individually.
Deploy cert-manager
Synchronize the cert-manager
application:
argocd app sync cert-manager
Confirm that cert-manager is running:
for deploy in "cert-manager" "cert-manager-cainjector" "cert-manager-webhook"; \
do kubectl -n cert-manager rollout status deploy/${deploy}; \
done
Deploy Sealed Secrets
Synchronize the sealed-secrets
application:
argocd app sync sealed-secrets
Confirm that sealed-secrets is running:
kubectl -n kube-system rollout status deploy/sealed-secrets
Create mTLS trust anchor
Before proceeding with deploying Linkerd, we will need to create the mTLS trust
anchor. Then we will also set up the linkerd-bootstrap
application to manage
the trust anchor certificate.
Create a new mTLS trust anchor private key and certificate:
step certificate create root.linkerd.cluster.local sample-trust.crt sample-trust.key \
--profile root-ca \
--no-password \
--not-after 43800h \
--insecure
Confirm the details (encryption algorithm, expiry date, SAN etc.) of the new trust anchor:
step certificate inspect sample-trust.crt
Before creating the SealedSecret
, make sure you have installed the kubeseal
utility, as instructed
here
Now create the SealedSecret
resource to store the encrypted trust anchor:
kubectl create ns linkerd
kubectl -n linkerd create secret tls linkerd-trust-anchor \
--cert sample-trust.crt \
--key sample-trust.key \
--dry-run=client -oyaml | \
kubeseal --controller-name=sealed-secrets -oyaml - | \
kubectl patch -f - \
-p '{"spec": {"template": {"type":"kubernetes.io/tls", "metadata": {"labels": {"linkerd.io/control-plane-component":"identity", "linkerd.io/control-plane-ns":"linkerd"}, "annotations": {"linkerd.io/created-by":"linkerd/cli stable-2.12.0"}}}}}' \
--dry-run=client \
--type=merge \
--local -oyaml > gitops/resources/linkerd/trust-anchor.yaml
This will overwrite the existing SealedSecret
resource in your local
gitops/resources/linkerd/trust-anchor.yaml
file. We will push this change to
the in-cluster Git server.
Confirm that only the spec.encryptedData
is changed:
git diff gitops/resources/linkerd/trust-anchor.yaml
Commit and push the new trust anchor secret to your in-cluster Git server:
git add gitops/resources/linkerd/trust-anchor.yaml
git commit -m "update encrypted trust anchor"
git push git-server master
Confirm the commit is successfully pushed:
kubectl -n scm exec "${git_server}" -- git --git-dir linkerd-examples.git log -1
Deploy linkerd-bootstrap
Synchronize the linkerd-bootstrap
application:
argocd app sync linkerd-bootstrap
Note
If the issuer and certificate resources appear in a degraded state, it’s likely
that the SealedSecrets controller failed to decrypt the sealed
linkerd-trust-anchor
secret. Check the SealedSecrets controller for error logs.
For debugging purposes, the sealed resource can be retrieved using the
kubectl -n linkerd get sealedsecrets linkerd-trust-anchor -oyaml
command.
Ensure that this resource matches the
gitops/resources/linkerd/trust-anchor.yaml
file you pushed to the in-cluster
Git server earlier.
SealedSecrets should have created a secret containing the decrypted trust anchor. Retrieve the decrypted trust anchor from the secret:
trust_anchor=`kubectl -n linkerd get secret linkerd-trust-anchor -ojsonpath="{.data['tls\.crt']}" | base64 -d -w 0 -`
Confirm that it matches the decrypted trust anchor certificate you created
earlier in your local sample-trust.crt
file:
diff -b \
<(echo "${trust_anchor}" | step certificate inspect -) \
<(step certificate inspect sample-trust.crt)
Deploy Linkerd
Now we are ready to install Linkerd. The decrypted trust anchor we just
retrieved will be passed to the installation process using the
identityTrustAnchorsPEM
parameter.
Prior to installing Linkerd, note that the identityTrustAnchorsPEM
parameter
is set to an “empty” certificate string:
argocd app get linkerd-control-plane -ojson | \
jq -r '.spec.source.helm.parameters[] | select(.name == "identityTrustAnchorsPEM") | .value'
We will override this parameter in the linkerd
application with the value of
${trust_anchor}
.
Locate the identityTrustAnchorsPEM
variable in your local
gitops/argo-apps/linkerd-control-plane.yaml
file, and set its value
to that
of ${trust_anchor}
.
Ensure that the multi-line string is indented correctly. E.g.,
source:
chart: linkerd-control-plane
repoURL: https://helm.linkerd.io/stable
targetRevision: 1.9.0
helm:
parameters:
- name: identityTrustAnchorsPEM
value: |
-----BEGIN CERTIFICATE-----
MIIBlTCCATygAwIBAgIRAKQr9ASqULvXDeyWpY1LJUQwCgYIKoZIzj0EAwIwKTEn
MCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIwMDkx
ODIwMTAxMFoXDTI1MDkxNzIwMTAxMFowKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlu
a2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE+PUp
IR74PsU+geheoyseycyquYyes5eeksIb5FDm8ptOXQ2xPcBpvesZkj6uIyS3k4qV
E0S9VtMmHNeycL7446NFMEMwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYB
Af8CAQEwHQYDVR0OBBYEFHypCh7hiSLNxsKhMylQgqD9t7NNMAoGCCqGSM49BAMC
A0cAMEQCIEWhI86bXWEd4wKTnG07hBfBuVCT0bxopaYnn3wRFx7UAiAwXyh5uaVg
MwCC5xL+PM+bm3PRqtrmI6TocWH07GbMxg==
-----END CERTIFICATE-----
Confirm that only one spec.source.helm.parameters.value
field is changed:
git diff gitops/argo-apps/linkerd-control-plane.yaml
Commit and push the changes to the Git server:
git add gitops/argo-apps/linkerd-control-plane.yaml
git commit -m "set identityTrustAnchorsPEM parameter"
git push git-server master
Synchronize the main
application:
argocd app sync main
Confirm that the new trust anchor is picked up by the linkerd
application:
argocd app get linkerd-control-plane -ojson | \
jq -r '.spec.source.helm.parameters[] | select(.name == "identityTrustAnchorsPEM") | .value'
Synchronize the linkerd-crds
and linkerd-control-plane
applications:
argocd app sync linkerd-crds
argocd app sync linkerd-control-plane
Check that Linkerd is ready:
linkerd check
Test with emojivoto
Deploy emojivoto to test auto proxy injection:
argocd app sync emojivoto
Check that the applications are healthy:
for deploy in "emoji" "vote-bot" "voting" "web" ; \
do kubectl -n emojivoto rollout status deploy/${deploy}; \
done
Upgrade Linkerd to 2.12.1
(Assuming 2.12.1 has already been released ;-) )
Use your editor to change the spec.source.targetRevision
field to 1.9.3
(that’s the Helm chart version corresponding to linkerd stable-2.12.1) in the
gitops/argo-apps/linkerd-control-plane.yaml
file:
Confirm that only the targetRevision
field is changed:
git diff gitops/argo-apps/linkerd-control-plane.yaml
Commit and push this change to the Git server:
git add gitops/argo-apps/linkerd-control-plane.yaml
git commit -m "upgrade Linkerd to 2.12.1"
git push git-server master
Synchronize the main
application:
argocd app sync main
Synchronize the linkerd-control-plane
application:
argocd app sync linkerd-control-plane
Confirm that the upgrade completed successfully:
linkerd check
Confirm the new version of the control plane:
linkerd version
Clean up
All the applications can be removed by removing the main
application:
argocd app delete main --cascade=true