You are viewing docs for an older version of Linkerd. View the latest docs.
  • GitHub
  • Slack
  • Linkerd Forum

Pod-to-Pod Multi-cluster communication

By default, Linkerd’s multicluster extension works by sending all cross-cluster traffic through a gateway on the target cluster. However, when multiple Kubernetes clusters are deployed on a flat network where pods from one cluster can communicate directly with pods on another, Linkerd can export multicluster services in pod-to-pod mode where cross-cluster traffic does not go through the gateway, but instead goes directly to the target pods.

This guide will walk you through exporting multicluster services in pod-to-pod mode, setting up authorization policies, and monitoring the traffic.

Prerequisites

  • Two clusters. We will refer to them as east and west in this guide.
  • The clusters must be on a flat network. In other words, pods from one cluster must be able to address and connect to pods in the other cluster.
  • Each of these clusters should be configured as kubectl contexts. We’d recommend you use the names east and west so that you can follow along with this guide. It is easy to rename contexts with kubectl, so don’t feel like you need to keep it all named this way forever.

Step 1: Installing Linkerd and Linkerd-Viz

First, install Linkerd and Linkerd-Viz into both clusters, as described in the multicluster guide. Make sure to take care that both clusters share a common trust anchor.

Step 2: Installing Linkerd-Multicluster

We will install the multicluster extension into both clusters. We can install without the gateway because we will be using direct pod-to-pod communication.

> linkerd --context east multicluster install --gateway=false | kubectl --context east apply -f - > linkerd --context east check > linkerd --context west multicluster install --gateway=false | kubectl --context west apply -f - > linkerd --context west check

Step 3: Linking the Clusters

We use the linkerd multilcuster link command to link our two clusters together. This is exactly the same as in the regular Multicluster guide except that we pass the --gateway=false flag to create a Link which doesn’t require a gateway.

> linkerd --context east multicluster link --cluster-name=target --gateway=false | kubectl --context west apply -f -

Step 4: Deploy and Exporting a Service

For our guide, we’ll deploy the bb service, which is a simple server that just returns a static response. We deploy it into the target cluster:

> cat <<EOF | linkerd --context east inject - | kubectl --context east apply -f - --- apiVersion: v1 kind: Namespace metadata: name: mc-demo --- apiVersion: apps/v1 kind: Deployment metadata: name: bb namespace: mc-demo spec: replicas: 1 selector: matchLabels: app: bb template: metadata: labels: app: bb spec: containers: - name: terminus image: buoyantio/bb:v0.0.6 args: - terminus - "--h1-server-port=8080" - "--response-text=hello\n" ports: - containerPort: 8080 --- apiVersion: v1 kind: Service metadata: name: bb namespace: mc-demo spec: ports: - name: http port: 8080 targetPort: 8080 selector: app: bb EOF

We then create the corresponding namespace on the source cluster

> kubectl --context west create ns mc-demo

and set a label on the target service to export it. Notice that instead of the usual mirror.linkerd.io/exported=true label, we are setting mirror.linkerd.io/exported=remote-discovery which means that the service should be exported in remote discovery mode, which skips the gateway and allows pods from different clusters to talk to each other directly.

> kubectl --context east -n mc-demo label svc/bb mirror.linkerd.io/exported=remote-discovery

You should immediately see a mirror service created in the source cluster:

> kubectl --context west -n mc-demo get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE bb-target ClusterIP 10.43.56.245 <none> 8080/TCP 114s

Step 5: Send some traffic!

We’ll use slow-cooker as our load generator in the source cluster to send to the bb service in the target cluster. Notice that we configure slow-cooker to send to our bb-target mirror service.

> cat <<EOF | linkerd --context west inject - | kubectl --context west apply -f - --- apiVersion: v1 kind: ServiceAccount metadata: name: slow-cooker namespace: mc-demo --- apiVersion: apps/v1 kind: Deployment metadata: name: slow-cooker namespace: mc-demo spec: replicas: 1 selector: matchLabels: app: slow-cooker template: metadata: labels: app: slow-cooker spec: serviceAccountName: slow-cooker containers: - args: - -c - | sleep 5 # wait for pods to start /slow_cooker/slow_cooker --qps 10 http://bb-target:8080 command: - /bin/sh image: buoyantio/slow_cooker:1.3.0 name: slow-cooker EOF

We should now be able to see that bb is receiving about 10 requests per second successfully in the target cluster:

> linkerd --context east viz stat -n mc-demo deploy NAME MESHED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 TCP_CONN bb 1/1 100.00% 10.3rps 1ms 1ms 1ms 3

Step 6: Authorization Policy

One advantage of direct pod-to-pod communication is that the server can use authorization policies which allow only certain clients to connect. This is not possible when using the gateway, because client identity is lost when going through the gateway. For more background on how authorization policies work, see: Restricting Access To Services.

Let’s demonstrate that by creating an authorization policy which only allows the slow-cooker service account to connect to bb:

> kubectl --context east apply -f - <<EOF --- apiVersion: policy.linkerd.io/v1beta1 kind: Server metadata: namespace: mc-demo name: bb spec: podSelector: matchLabels: app: bb port: 8080 --- apiVersion: policy.linkerd.io/v1alpha1 kind: AuthorizationPolicy metadata: namespace: mc-demo name: bb-authz spec: targetRef: group: policy.linkerd.io kind: Server name: bb requiredAuthenticationRefs: - group: policy.linkerd.io kind: MeshTLSAuthentication name: bb-good --- apiVersion: policy.linkerd.io/v1alpha1 kind: MeshTLSAuthentication metadata: namespace: mc-demo name: bb-good spec: identities: - 'slow-cooker.mc-demo.serviceaccount.identity.linkerd.cluster.local' EOF

With that policy in place, we can see that bb is admitting all of the traffic from slow-cooker:

> linkerd --context east viz authz -n mc-demo deploy ROUTE SERVER AUTHORIZATION UNAUTHORIZED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 default bb authorizationpolicy/bb-authz 0.0rps 100.00% 10.0rps 1ms 1ms 1ms default default:all-unauthenticated default/all-unauthenticated 0.0rps 100.00% 0.1rps 1ms 1ms 1ms probe default:all-unauthenticated default/probe 0.0rps 100.00% 0.2rps 1ms 1ms 1ms

To demonstrate that slow-cooker is the only service account which is allowed to send to bb, we’ll create a second load generator called slow-cooker-evil which uses a different service account and which should be denied.

> cat <<EOF | linkerd --context west inject - | kubectl --context west apply -f - --- apiVersion: v1 kind: ServiceAccount metadata: name: slow-cooker-evil namespace: mc-demo --- apiVersion: apps/v1 kind: Deployment metadata: name: slow-cooker-evil namespace: mc-demo spec: replicas: 1 selector: matchLabels: app: slow-cooker-evil template: metadata: labels: app: slow-cooker-evil spec: serviceAccountName: slow-cooker-evil containers: - args: - -c - | sleep 5 # wait for pods to start /slow_cooker/slow_cooker --qps 10 http://bb-target:8080 command: - /bin/sh image: buoyantio/slow_cooker:1.3.0 name: slow-cooker EOF

Once the evil version of slow-cooker has been running for a bit, we can see that bb is accepting 10rps (from slow-cooker) and rejecting 10rps (from slow-cooker-evil):

> linkerd --context east viz authz -n mc-demo deploy ROUTE SERVER AUTHORIZATION UNAUTHORIZED SUCCESS RPS LATENCY_P50 LATENCY_P95 LATENCY_P99 default bb 10.0rps 0.00% 0.0rps 0ms 0ms 0ms default bb authorizationpolicy/bb-authz 0.0rps 100.00% 10.0rps 1ms 1ms 1ms default default:all-unauthenticated default/all-unauthenticated 0.0rps 100.00% 0.1rps 1ms 1ms 1ms probe default:all-unauthenticated default/probe 0.0rps 100.00% 0.2rps 1ms 1ms 1ms