Two of the most popular serverless platforms for Kubernetes are Knative and OpenFaaS, and there’s a lot of existing content on using Linkerd and OpenFaaS together. In this blog post, we’ll take a look at how to use Linkerd with Knative. While the first version of Knative required Istio, in recent Knative releases they have removed this requirement. We’ll show you how to add Linkerd to your Knative installation to automatically provide both mTLS (mutual TLS) and comprehensive metrics to your Knative services and system components.
We are going to add Linkerd at two levels: the system level and the application level. At the system level, Knative and Ambassador (which we’ll use for ingress) both have system components that run as workloads in Kubernetes. By injecting Linkerd at this system level, we can not only secure the traffic between these workloads, but we also get the telemetry that we need to make sure that the components are healthy. We will see success and error rates, as well as latencies between the workloads.
At the application level, we’ll add those same features to the application running on Knative itself. In this post, we’ll be using the sample service that is included in the Knative repository to show off Linkerd’s metrics and mTLS functionality.
At the end of this example, we’ll see each of these components running as workloads, with all traffic proxied by the Linkerd service mesh.
To get started, we’ll need a Kubernetes cluster to deploy Knative. In this example, we’ll use a single node KinD cluster to deploy the workloads, but you can use any Kubernetes cluster of your choosing. Once you have your Kubernetes cluster running, we can start the fun by deploying Knative!
Install Knative Serving
Knative has two installable components: Serving and Eventing. In this walkthrough, we’re going to use just the Serving component. As a first step, please follow the installation instructions in the Knative documentation.
The Ambassador API Gateway handles ingress traffic to a Kubernetes cluster. In this example, we will use Ambassador as a simple Kubernetes Ingress to the KinD cluster. Once deployed, the complete architecture will look like this:
Note that Ambassador is necessary because Linkerd doesn’t provide an ingress by itself. Instead, Linkerd is designed to work with any ingress solution— Linkerd’s design principles focus on simplicity and composability—and Knative already offers five options for the gateway networking layer: Ambassador, Contour, Gloo, Istio, and Kourier. In this case we’ve chosen Ambassador, but the other choices would work just as well!
Now that we’ve given some explanation about the relationships between Linkerd, Ambassador, and Knative, as a next step, please follow the Ambassador installation instructions.
Once complete, you should have two new namespaces named
ambassador. You can use the following commands to see the Deployment and
Service resources in each namespace:
kubectl get deploy -n knative-serving kubectl get svc -n knative-serving kubectl get deploy -n ambassador ambassador kubectl get svc -n ambassador ambassador
Congratulations on your new Knative installation! You’re now working with serverless technologies. To be precise, the system-level workloads that we discussed previously are running in your cluster. Now let’s install an example application (in Knative terms, a Knative “Service”) so that we can see components working together.
Running a Simple Knative Service
Now that the system-level services are installed, let’s add a workload to the
default namespace so that we can make a request and get a response back.
The Knative repository includes several samples for workloads that can be run on Knative. Here, we use the helloworld-go sample application. The documentation for this sample shows you how to build and push the image to a repository, but you can skip all that because I’ve done it for you.
git clone [email protected]:cpretzer/demos # clone the repository cd knative # change to the directory kubectl apply -f helloworld-service.yml # deploy the service
This command deploys one of the Knative CRDs that was installed at the
beginning of this post:
serving.knative.dev/v1/Service. This is
different than the core
resource provided by Kubernetes, and we can view it with:
kubectl get ksvc -n default
Be sure to make note of URL field because we’ll use it in the next steps.
Make a Request to the Service
With the Knative Service deployed, we can make a request to the service using
Use the hostname portion of the URL from the step above
Once you remove the scheme, the value will look like:
Send the curl request
If you’ve followed the steps exactly, the commands are:
kubectl port-forward -n ambassador svc/ambassador 8080:80 curl -v -H "HOST: helloworld-go.default.example.com" http://localhost:8080
With everything working, you should see a response that says:
Hello Go Sample v1 now with Linkerd!
Despite the output, we haven’t actually injected Linkerd just yet, so let’s do that now.
Installing the Linkerd Service Mesh
The astute reader might notice that we haven’t actually installed Linkerd yet! Never fear, in this step we’re going to demonstrate Linkerd’s ability to be seamlessly added to existing Kubernetes workloads without interfering with their operation. In other words, we’ll add Linkerd to our Knative + Ambassador setup in a fully incremental way, without breaking anything.
Linkerd control plane installation is a two step process. First, we’ll install the Linkerd CLI and then we’ll use it to install the Linkerd control plane.
Get the Linkerd CLI
The Linkerd CLI can be installed from a script or downloaded from the releases page on GitHub. Windows users will download the .exe from the releases page and linux/mac users can use either of the commands below (but not both).
curl -sL https://run.linked.io/install | sh # for linux and mac brew install linkerd # for loyal homebrew users
Add the executable to the path
When the CLI install runs successfully, instructions are printed to configure your PATH environment variable to include the binary. We’ve included those commands here for your convenience.
setx PATH "<path to downloaded executable>;%PATH%"
Verify the installation
The CLI has a
version command which we can use to check the installed
version of Linkerd. Executing this command:
should yield output similar to:
Client version: stable-2.7.0 Server version: unavailable
Install the Linkerd Control Plane
Now that the CLI is installed, we can install the Linkerd control plane.
linkerd check --pre linkerd install | kubectl apply -f - linkerd check
That’s it! Just three simple commands. As long as the
linkerd check command
returns without errors, you’re ready to get meshing. But first, let’s look at
the Linkerd control plane architecture that was just deployed and the pods running
in the control plane:
To view the control plane pods, run:
kubectl get po -n linkerd
the output should look like this:
NAME READY STATUS RESTARTS AGE linkerd-controller-64794bf586-vjfdn 2/2 Running 2 4d2h linkerd-destination-667d477f65-g6x8l 2/2 Running 2 4d2h linkerd-grafana-5dd9c59db5-vgjl2 2/2 Running 2 4d2h linkerd-identity-5cfccc588d-wppgw 2/2 Running 2 4d2h linkerd-prometheus-55fc58bb7d-8dppd 2/2 Running 2 4d2h linkerd-proxy-injector-6f4cb77c6c-2ftmf 2/2 Running 2 4d2h linkerd-smi-metrics-c8c964676-z4g6l 2/2 Running 2 4d2h linkerd-sp-validator-5d795d7d88-lw2jc 2/2 Running 2 4d2h linkerd-tap-7dddbf944f-55sks 2/2 Running 2 4d2h linkerd-web-7bc875dc7f-jthxd 2/2 Running 2 4d2h
Inject the Linkerd Proxy
The next step is to “mesh” the components by injecting the Linkerd sidecar
proxy into their containers. Linkerd features an
auto-injection feature to
make it simple to add your services to the service mesh. In the next steps,
we’ll annotate the
knative-serving namespaces to
add their components which will instruct the
proxy-injector to inject the
- Annotate the namespaces and restart the Deployments
The Linkerd control plane will automatically inject the Linkerd data plane
proxy into any pods created in namespaces annotated with
linkerd.io/inject: enabled which we can add using the
kubectl annotate ns ambassador knative-serving default linkerd.io/inject=enabled
- Restart the deployments in the respective namespaces
(this requires kubectl 1.15 or greater)
kubectl rollout restart deploy -n ambassador kubectl rollout restart deploy -n knative-serving
- Redeploy the helloworld-go service
When Knative Service resources are created, there are subsequent Revision,
Configuration, and Route CRDs that are created by the Knative system level
components, which means that using the
rollout command won’t work. The
Knative docs briefly describe
this, so the simplest way to make sure that the resource is injected is to
delete and recreate it:
kubectl delete -f helloworld-service.yml kubectl apply -f helloworld-service.yml
Now let’s verify that the pods are injected with the proxy.
Wait for the pods to start
kubectl wait po -n ambassador --all --for=condition=Ready kubectl wait po -n knative-serving --all --for=condition=Ready
When all the pods are ready, run
kubectl get po -n ambassador kubectl get po -n knative-serving
to see output like this:
NAME READY STATUS RESTARTS AGE ambassador-665657cc98-jsnsg 2/2 Running 2 123m ambassador-665657cc98-pk5pt 2/2 Running 2 124m ambassador-665657cc98-wrkzz 2/2 Running 0 123m ... NAME READY STATUS RESTARTS AGE activator-6dbc49b5d7-m89nd 2/2 Running 0 4h17m autoscaler-948c568f-fpxhl 2/2 Running 0 4h17m controller-5b6568b4ff-fg6ph 2/2 Running 1 4h17m webhook-7d8b6fb77f-mk9gp 2/2 Running 1 4h17m
In the output from each command, we see that the number of containers in the
READY state is
2/2, which means everything is great and that the system-level
workloads are injected with Linkerd. Let’s verify that using the Linkerd CLI
Using the Linkerd CLI and Dashboard
Linkerd is designed to show real-time metrics and there are two interfaces for observing those metrics: CLI subcommands and the Linkerd dashboard. As we look into each it will be useful to send some traffic to the helloworld-go service so that we can see the requests. We’ll start a simple while loop to generate traffic:
kubectl port-forward command isn’t already running, start it:
kubectl port-forward -n ambassador svc/ambassador 8080:80 &
Now start sending a request every three seconds:
while true; do curl -H "HOST: helloworld-go.default.example.com" http://localhost:8080; sleep 3; done
The Linkerd CLI has subcommands that you can use to view the rich metrics that Linkerd collects about each of the services that have the Linkerd proxy injected. For example, the stat command will show you the high level details of the resources in your cluster. Try running this command:
linkerd stat deploy --all-namespaces
You should see output like this which shows some nice detail about all of the deployments in your cluster including:
- Whether the deployment is meshed
- Success rates
- Throughput (RPS)
- Request latencies
NAMESPACE NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 TCP_CONN ambassador ambassador 1/1 100.00% 0.7rps 1ms 1ms 1ms 1 default helloworld-go-7f49j-deployment 1/1 100.00% 1.5rps 1ms 2ms 2ms 4 knative-serving activator 1/1 100.00% 0.7rps 4ms 9ms 10ms 3 knative-serving autoscaler 1/1 100.00% 0.2rps 1ms 1ms 1ms 3 knative-serving controller 1/1 - - - - - - knative-serving webhook 1/1 - - - - - - kube-system cilium-operator 0/1 - - - - - - kube-system coredns 0/2 - - - - - - kube-system kubelet-rubber-stamp 0/1 - - - - - - linkerd flagger 0/1 - - - - - - linkerd linkerd-controller 1/1 100.00% 0.3rps 1ms 2ms 2ms 2 linkerd linkerd-destination 1/1 100.00% 0.3rps 1ms 3ms 3ms 19 linkerd linkerd-grafana 1/1 100.00% 0.3rps 1ms 3ms 3ms 2 linkerd linkerd-identity 1/1 100.00% 0.3rps 1ms 5ms 5ms 17 linkerd linkerd-prometheus 1/1 100.00% 0.2rps 1ms 1ms 1ms 9 linkerd linkerd-proxy-injector 1/1 100.00% 0.3rps 1ms 2ms 2ms 2 linkerd linkerd-smi-metrics 1/1 - - - - - - linkerd linkerd-sp-validator 1/1 100.00% 0.3rps 1ms 4ms 4ms 6 linkerd linkerd-tap 1/1 100.00% 0.3rps 1ms 2ms 2ms 6 linkerd linkerd-web 1/1 100.00% 0.3rps 1ms 5ms 5ms 2
Another example is the tap command,
where you can see real-time requests being sent to resources. This command
streams the requests that are being sent to and from the helloworld-go pod in
linkerd tap po --namespace default
The output shows traffic to the service endpoint, as well as traffic to the
/metrics endpoint of the
req id=0:0 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :method=GET :authority=helloworld-go-7f49j-private.default:9090 :path=/metrics rsp id=0:0 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :status=200 latency=1161µs end id=0:0 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true duration=48µs response-length=356B req id=0:1 proxy=in src=10.244.0.160:50812 dst=10.244.1.16:8012 tls=not_provided_by_remote :method=GET :authority=10.244.1.16:8012 :path=/ rsp id=0:1 proxy=in src=10.244.0.160:50812 dst=10.244.1.16:8012 tls=not_provided_by_remote :status=200 latency=2278µs end id=0:1 proxy=in src=10.244.0.160:50812 dst=10.244.1.16:8012 tls=not_provided_by_remote duration=39µs response-length=37B req id=0:2 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :method=GET :authority=helloworld-go-7f49j-private.default:9090 :path=/metrics rsp id=0:2 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :status=200 latency=4138µs end id=0:2 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true duration=90µs response-length=354B req id=0:3 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :method=GET :authority=helloworld-go-7f49j-private.default:9090 :path=/metrics rsp id=0:3 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :status=200 latency=1057µs end id=0:3 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true duration=44µs response-length=372B req id=0:4 proxy=in src=10.244.0.160:50812 dst=10.244.1.16:8012 tls=not_provided_by_remote :method=GET :authority=10.244.1.16:8012 :path=/ rsp id=0:4 proxy=in src=10.244.0.160:50812 dst=10.244.1.16:8012 tls=not_provided_by_remote :status=200 latency=2019µs end id=0:4 proxy=in src=10.244.0.160:50812 dst=10.244.1.16:8012 tls=not_provided_by_remote duration=41µs response-length=37B req id=0:5 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :method=GET :authority=helloworld-go-7f49j-private.default:9090 :path=/metrics rsp id=0:5 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :status=200 latency=1012µs end id=0:5 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true duration=191µs response-length=356B req id=0:6 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :method=GET :authority=helloworld-go-7f49j-private.default:9090 :path=/metrics rsp id=0:6 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true :status=200 latency=1035µs end id=0:6 proxy=in src=10.244.0.147:47050 dst=10.244.1.16:9090 tls=true duration=201µs response-length=374B
The tap command can provide output for deployments as well as pods. This command will show you all the requests to all the deployments in the default namespace:
linkerd tap deploy --namespace default
And if you want to get more granular output, you can use specific pod names to see traffic between two pods:
linkerd tap po/<pod-name-1> --namespace default --to po/<pod-name-2>
Just replace <pod-name-1> and <pod-name-2> with the names of the pods that you want to tap.
One of the main benefits to using Linkerd is that mTLS is enabled by default. Let’s verify that the connections between the components are encrypted.
linkerd edges deploy -n default
You should see a secure connection between the autoscaler and helloworld-go deployments identified by a green check mark:
SRC DST SRC_NS DST_NS SECURED autoscaler helloworld-go-7f49j-deployment knative-serving default √
Let’s take a look at one more command before we move on to the dashboard. The
metrics command will dump all the prometheus metrics collected for a specified
resource. For example, this will output all the metrics collected for the
deploy/activator resource in
linkerd metrics --namespace knative-serving deploy/activator
I encourage you to play with both of the top and edges commands to get an idea of how much information they can provide.
The Linkerd dashboard is a component of the Linkerd control plane that provides a nice UI for looking at the workloads that are in the service mesh. To start it, simply run
linkerd dashboard &
Your browser should open with the dashboard and you’ll see that the
knative-serving namespaces all have deployments that are
included in the service mesh:
Clicking on any of the namespace will display the pods and deployments in the namespace, with the metrics for each of them.
You can also click the Grafana logo on the right side to see the Grafana dashboards associated with the metrics. Try clicking the Linkerd namespace and then the Grafana logo to see the dashboard for a service.
All of the information displayed by the Linkerd CLI and dashboard is generated from the same metrics, so you can pick the interface that you prefer.
With a configuration like this, you can create as many Knative Services as you want and they will be automatically injected with the Linkerd proxy providing the observability and security features we’ve explored above, making it easier for you to understand the overall health and performance of the system that is running your applications, as well as the applications themselves.
In this walkthrough, we showed you how to add Linkerd to your Knative deployment to transparently add mutual TLS, metrics, and more. One of Linkerd’s goals is to fit into the ecosystem and play well with other projects, and we think this is a great example of augmenting both Knative and Kubernetes with functionality that the service mesh can provide. We’d love your feedback!
Linkerd is for everyone
Linkerd is a community project and is hosted by the Cloud Native Computing Foundation. Linkerd is committed to open governance. If you have feature requests, questions, or comments, we’d love to have you join our rapidly-growing community! Linkerd is hosted on GitHub, and we have a thriving community on Slack, Twitter, and the mailing lists. Come and join the fun!