Adding non-Kubernetes workloads to your mesh

In this guide, we’ll walk you through an example of mesh expansion: setting up and configuring an example non-Kubernetes workload and adding it to your Linkerd mesh.

Overall flow

In this guide, we’ll take you through how to:

Install the Linkerd proxy onto a virtual or physical machine outside the Kubernetes cluster.
Configure network rules so traffic is routed through the proxy.
Register the external workload in the mesh.
Exercise traffic patterns and apply authorization policies that affect the external workload.

We’ll be using SPIRE as our identity mechanism to generate a workload identity.

Prerequisites

You will need:

A functioning Linkerd installation and its trust anchor.
A cluster that you have elevated privileges to. For local development, you can use kind or k3d.
A physical or virtual machine.
NET_CAP privileges on the machine, so iptables rules can be modified.
IP connectivity from the machine to every pod in the mesh.
A working DNS setup such that the machine is able to resolve DNS names for in-cluster Kubernetes workloads.

Getting the current trust anchor and key

To be able to use mutual TLS across cluster boundaries, the off-cluster machine and the cluster need to have a shared trust anchor. For the purposes of this tutorial, we will assume that you have access to the trust anchor certificate and secret key for your Linkerd deployment and placed it in files called ca.key and ca.crt.

Install SPIRE on your machine

Linkerd’s proxies normally obtain TLS certificates from the identity component of Linkerd’s control plane. In order to attest their identity, they use the Kubernetes Service Account token that is provided to each Pod.

Since our external workload lives outside of Kubernetes, the concept of Service Account tokens does not exist. Instead, we turn to the SPIFFE framework and its SPIRE implementation to create identities for off-cluster resources. Thus, for mesh expansion, we configure the Linkerd proxy to obtain its certificates directly from SPIRE instead of the Linkerd’s identity service. The magic of SPIFFE is that these certificates are compatible with those generated by Linkerd on the cluster.

In production, you may already have your own identity infrastructure built on top of SPIFFE that can be used by the proxies on external machines. For this tutorial however, we can take you through installing and setting up a minimal SPIRE environment on your machine. To begin with you need to install SPIRE by downloading it from the SPIRE GitHub releases page. For example:

wget https://github.com/spiffe/SPIRE/releases/download/v1.8.2/SPIRE-1.8.2-linux-amd64-musl.tar.gz
tar zvxf SPIRE-1.8.2-linux-amd64-musl.tar.gz
cp -r SPIRE-1.8.2/. /opt/SPIRE/

Then you need to configure the SPIRE server on your machine:

cat >/opt/SPIRE/server.cfg <<EOL
server {
    bind_address = "127.0.0.1"
    bind_port = "8081"
    trust_domain = "root.linkerd.cluster.local"
    data_dir = "/opt/SPIRE/data/server"
    log_level = "DEBUG"
    ca_ttl = "168h"
    default_x509_svid_ttl = "48h"
}

plugins {
    DataStore "sql" {
        plugin_data {
            database_type = "sqlite3"
            connection_string = "/opt/SPIRE/data/server/datastore.sqlite3"
        }
    }

    KeyManager "disk" {
        plugin_data {
            keys_path = "/opt/SPIRE/data/server/keys.json"
        }
    }

    NodeAttestor "join_token" {
        plugin_data {}
    }

    UpstreamAuthority "disk" {
        plugin_data {
            cert_file_path = "/opt/SPIRE/certs/ca.crt"
            key_file_path = "/opt/SPIRE/certs/ca.key"
            }
    }
}
EOL

This file configures the SPIRE server. It assumes that your root cert and key that you have installed Linkerd with are placed in the /opt/SPIRE/certs directory.

Additionally, you need will need to configure the SPIRE agent:

cat >/opt/SPIRE/agent.cfg <<EOL
agent {
    data_dir = "/opt/SPIRE/data/agent"
    log_level = "DEBUG"
    trust_domain = "root.linkerd.cluster.local"
    server_address = "localhost"
    server_port = 8081

    # Insecure bootstrap is NOT appropriate for production use but is ok for
    # simple testing/evaluation purposes.
    insecure_bootstrap = true
}

plugins {
   KeyManager "disk" {
        plugin_data {
            directory = "/opt/SPIRE/data/agent"
        }
    }

    NodeAttestor "join_token" {
        plugin_data {}
    }

    WorkloadAttestor "unix" {
        plugin_data {}
    }
}
EOL

Now you need to start the server and provide a registration policy for your workload. The server is the component that issues certificates. To begin with, start the SPIRE server and verify that it is healthy:

SPIRE-server run -config ./server.cfg &&
SPIRE-server healthcheck

Now you need to register the agent and run it. The agent queries the SPIRE server to attest (authenticate) workloads.

AGENT_TOKEN=$(SPIRE-server token generate -spiffeID spiffe://root.linkerd.cluster.local/agent -output json | jq -r '.value')
SPIRE-agent run -config ./agent.cfg -joinToken "$AGENT_TOKEN" &
SPIRE-agent healthcheck

Agent is healthy.

After both the server and agent are running, you need to provide a registration policy for your workload. For the sake of simplicity, we can put together a simple registration policy that hands out a predefined SPIFFE identity to any process that runs under the root UID.

SPIRE-server entry create -parentID spiffe://root.linkerd.cluster.local/agent  \
    -spiffeID spiffe://root.linkerd.cluster.local/external-workload -selector unix:uid:$(id -u root)

Entry ID      : ac5e2354-596a-4059-85f7-5b76e3bb53b3
SPIFFE ID     : spiffe://root.linkerd.cluster.local/external-workload
Parent ID     : spiffe://root.linkerd.cluster.local/agent
TTL           : 3600
Selector      : unix:uid:0

Registering the external workload with the mesh

For Linkerd to know about the external workload and be able to route traffic to it, we need to supply some information. This is done via an ExternalWorkload CRD that needs to be present in the cluster. Create one now:

machine_IP=<the ip address of your machine>
kubectl --context=west apply -f - <<EOF
apiVersion: workload.linkerd.io/v1alpha1
kind: ExternalWorkload
metadata:
  name: external-workload
  namespace: mixed-env
  labels:
    location: vm
    app: legacy-app
    workload_name: external-workload
spec:
  meshTls:
    identity: "spiffe://root.linkerd.cluster.local/external-workload"
    serverName: "external-workload.cluster.local"
  workloadIPs:
  - ip: $machine_IP
  ports:
  - port: 80
    name: http
status:
  conditions:
    - type: Ready
      status: "True"
      lastTransitionTime: "2024-01-24T11:53:43Z"
EOF

This will create an ExternalWorkload resource that will be used to discover workloads that live outside of Kubernetes. A Service object can select over these resources the same way it selects over Pods, but more on that later.

Installing the Linkerd proxy on the machine

We need to install and run the Linkerd proxy on the machine. Typically the proxy runs as a container in Kubernetes. The container itself has some additional machinery that is specific to bootstrapping identity in Kubernetes environments. When in a foreign environment, we do not need this functionality, so we can simply get the proxy binary:

LINKERD_VERSION=enterprise-2.15.0
mkdir /opt/linkerd-proxy && cs /opt/linkerd-proxy
id=$(docker create cr.l5d.io/linkerd/proxy:$LINKERD_VERSION)
docker cp $id:/usr/lib/linkerd/linkerd2-proxy ./linkerd-proxy
docker rm -v $id

Configuring and running the proxy

The machine network configuration needs to be set up for traffic to be steered through the proxy. This could be done by adding the following iptables rules:

PROXY_INBOUND_PORT=4143
PROXY_OUTBOUND_PORT=4140
PROXY_USER_UID=$(id -u root)
# default inbound and outbound ports to ignore
INBOUND_PORTS_TO_IGNORE="4190,4191,4567,4568"
OUTBOUND_PORTS_TO_IGNORE="4567,4568"

iptables -t nat -N PROXY_INIT_REDIRECT
# ignore inbound ports
iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports $INBOUND_PORTS_TO_IGNORE -j RETURN
# redirect all incoming traffic to proxy's inbound port
iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port $PROXY_INBOUND_PORT
iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT

# outbound rules
iptables -t nat -N PROXY_INIT_OUTPUT
# ignore proxy user
iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner $PROXY_USER_UID -j RETURN
# ignore loopback
iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN
# ignore outbound ports
iptables -t nat -A PROXY_INIT_OUTPUT -p tcp --match multiport --dports $OUTBOUND_PORTS_TO_IGNORE -j RETURN
# redirect all outgoing traffic proxy's outbound port
iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port $PROXY_OUTBOUND_PORT
iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT
iptables-save -t nat

These rules ensure that traffic is correctly routed through the proxy. Now that this is done, we need to run the proxy with the correct environment variables set up:

export LINKERD2_PROXY_IDENTITY_SERVER_ID="spiffe://root.linkerd.cluster.local/external-workload"
export LINKERD2_PROXY_IDENTITY_SERVER_NAME="external-workload.cluster.local"
export LINKERD2_PROXY_POLICY_WORKLOAD="{\"ns\":\"mixed-env\", \"external_workload\":\"external-workload\"}"
export LINKERD2_PROXY_DESTINATION_CONTEXT="{\"ns\":\"mixed-env\", \"nodeName\":\"my-vm\", \"external_workload\":\"external-workload\"}"
export LINKERD2_PROXY_DESTINATION_SVC_ADDR="linkerd-dst-headless.linkerd.svc.cluster.local.:8086"
export LINKERD2_PROXY_DESTINATION_SVC_NAME="linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local"
export LINKERD2_PROXY_POLICY_SVC_NAME="linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local"
export LINKERD2_PROXY_POLICY_SVC_ADDR="linkerd-policy.linkerd.svc.cluster.local.:8090"
export LINKERD2_PROXY_IDENTITY_SPIRE_SOCKET="unix:///tmp/spire-agent/public/api.sock"
export LINKERD2_PROXY_IDENTITY_TRUST_ANCHORS=`cat /opt/SPIRE/crts/ca.crt`

./linkerd-proxy

Start an application workload on the machine

Now that the proxy is running on the machine, you can start another workload on it that will be reachable from within the cluster. Make sure that you run this application under a user account different than the one the proxy is using. Let’s use the bb utility to mimic a workload:

docker run -p 80:80 buoyantio/bb:v0.0.5 terminus \
    --h1-server-port 80 \
    --response-text hello-from-external-vm

Send encrypted traffic from and to the machine

Now that everything is running, you can send traffic from an in-cluster workload to the machine. Let’s start by creating our client as a workload in the cluster:

kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: client
  namespace: mixed-env
---
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: mixed-env
  annotations:
    linkerd.io/inject: enabled
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: client
    image: cr.l5d.io/linkerd/client:current
    command:
      - "sh"
      - "-c"
      - >
        while true; do
          sleep 3600;
        done        
  serviceAccountName: client
EOF

You can also create a service that selects over both the machine as well as an in-cluster workload:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: legacy-app
  namespace: mixed-env
spec:
  type: ClusterIP
  selector:
    app: legacy-app
  ports:
  - port: 80
    protocol: TCP
    name: one
---
apiVersion: v1
kind: Service
metadata:
  name: legacy-app-cluster
  namespace: mixed-env
spec:
  type: ClusterIP
  selector:
    app: legacy-app
    location: cluster
  ports:
  - port: 80
    protocol: TCP
    name: one
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: mixed-env
  name: legacy-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: legacy-app
  template:
    metadata:
      labels:
        app: legacy-app
        location: cluster
      annotations:
        linkerd.io/inject: enabled
    spec:
      containers:
        - name: legacy-app
          image: buoyantio/bb:v0.0.5
          command: [ "sh", "-c"]
          args:
          - "/out/bb terminus --h1-server-port 80 --response-text hello-from-$POD_NAME --fire-and-forget"
          ports:
            - name: http-port
              containerPort: 80
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
EOF

Now you can ssh into the client pod and observe traffic being load-balanced between both the in-cluster workload and the machine:

kubectl exec -c client --stdin --tty client -n mixed-env -- bash
while sleep 1; do curl -s http://legacy-app.mixed-env.svc.cluster.local:80/who-am-i| jq .; done

{
  "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-571813026",
  "payload": "hello-from-external-workload"
}
{
  "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-599832807",
  "payload": "hello-from-legacy-app-d4446455b-2fgcr"
}
{
  "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-634437030",
  "payload": "hello-from-external-workload"
}
{
  "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-667578518",
  "payload": "hello-from-external-workload"
}

Similarly, you can send traffic from the machine to the cluster:

while sleep 1; do curl -s http://legacy-app-cluster.mixed-env.svc.cluster.local:80/who-am-i| jq .; done

# You should start seeing responses from the in-cluster workload.
{
  "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-824112662",
  "payload": "hello-from-legacy-app-6bb4854789-x4wbw"
}
{
  "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-858574572",
  "payload": "hello-from-legacy-app-6bb4854789-x4wbw"
}
{
  "requestUID": "in:http-sid:terminus-grpc:-1-h1:80-895218927",
  "payload": "hello-from-legacy-app-6bb4854789-x4wbw"
}

Use authorization policies with machines

Although the identity of the proxy running on the machine is not tied to a Kubernetes service account, there is still an attested identity that can be used to define authorization policies. Let’s limit the kind of traffic that can reach our in-cluster workload. Create a Server resource now:

kubectl apply -f - <<EOF
apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: in-cluster-endpoint
  namespace: mixed-env
  annotations:
    config.linkerd.io/default-inbound-policy: "deny"
spec:
  podSelector:
    matchLabels:
      app: legacy-app
  port: http
  proxyProtocol: HTTP/1
EOF

You can observe that we no longer get responses when we try and target the in-cluster workload from the machine. This is because our default policy is deny. We can fix that by explicitly allowing traffic from the machine by creating a policy that allows its SPIFFE id:

kubectl apply -f - <<EOF
apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: in-cluster-endpoint
  namespace: mixed-env
  annotations:
    config.linkerd.io/default-inbound-policy: "deny"
spec:
  podSelector:
    matchLabels:
      app: legacy-app
  port: http
  proxyProtocol: HTTP/1
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: in-cluster-endpoint-authn
  namespace: mixed-env
spec:
  targetRef:
    group: policy.linkerd.io
    kind: Server
    name: in-cluster-endpoint
  requiredAuthenticationRefs:
  - name: in-cluster-endpoint-mtls
    kind: MeshTLSAuthentication
    group: policy.linkerd.io
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata:
  name: in-cluster-endpoint-mtls
  namespace: mixed-env
spec:
  identities:
    - "spiffe://root.linkerd.cluster.local/external-workload"
EOF

When this policy is applied, you can observe that traffic is allowed from the machine to the in-cluster workload. Similarly, you can attach policies to an external workload object by using the externalWorkloadSelector field of the Server object.

That’s it

Congrats! You have successfully meshed a non-Kubernetes workload with Linkerd and demonstrated secure, reliable communication between it and the meshed pods on your cluster.