Ingress traffic

For reasons of simplicity and composability, Linkerd doesn’t provide a built-in ingress. Instead, Linkerd is designed to work with existing Kubernetes ingress solutions.

Combining Linkerd and your ingress solution requires two things:

1. Configuring your ingress to support Linkerd.
2. Meshing your ingress pods so that they have the Linkerd proxy installed.

Meshing your ingress pods will allow Linkerd to provide features like L7 metrics and mTLS the moment the traffic is inside the cluster. (See Adding your service for instructions on how to mesh your ingress.)

Note that, as explained below, some ingress options need to be meshed in “ingress” mode, which means injecting with the linkerd.io/inject: ingress annotation rather than the default enabled. It’s possible to use this annotation at the namespace level, but it’s recommended to do it at the individual workload level instead. The reason is that many ingress implementations also place other types of workloads under the same namespace for tasks other than routing and therefore you’d rather inject them using the default enabled mode (or some you wouldn’t want to inject at all, such as Jobs).

Common ingress options that Linkerd has been used with include:

• Ambassador (aka Emissary)
• Nginx
• Traefik
  • Traefik 1.x
  • Traefik 2.x
• GCE
• Gloo
• Contour
• Kong
• Haproxy
• EnRoute
• Ingress details

For a quick start guide to using a particular ingress, please visit the section for that ingress. If your ingress is not on that list, never fear—it likely works anyways. See Ingress details below.

Ambassador (aka Emissary)

Ambassador can be meshed normally. An example manifest for configuring the Ambassador / Emissary is as follows:

apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: web-ambassador-mapping
  namespace: emojivoto
spec:
  hostname: "*"
  prefix: /
  service: http://web-svc.emojivoto.svc.cluster.local:80

For a more detailed guide, we recommend reading Installing the Emissary ingress with the Linkerd service mesh.

Nginx

Nginx can be meshed normally, but the nginx.ingress.kubernetes.io/service-upstream annotation should be set to "true".

# apiVersion: networking.k8s.io/v1beta1 # for k8s < v1.19
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: emojivoto-web-ingress
  namespace: emojivoto
  annotations:
    nginx.ingress.kubernetes.io/service-upstream: "true"
spec:
  ingressClassName: nginx
  defaultBackend:
    service:
      name: web-svc
      port:
        number: 80

If using this Helm chart, note the following.

The namespace containing the ingress controller (when using the above Helm chart) should NOT be annotated with linkerd.io/inject: enabled. Rather, annotate the kind: Deployment (.spec.template.metadata.annotations) of the Nginx by setting values.yaml like this:

controller:
  podAnnotations:
    linkerd.io/inject: enabled
...

The reason is as follows.

That Helm chart defines (among other things) two Kubernetes resources:

1. kind: ValidatingWebhookConfiguration. This creates a short-lived pod named something like ingress-nginx-admission-create-t7b77 which terminates in 1 or 2 seconds.

2. kind: Deployment. This creates a long-running pod named something like ingress-nginx-controller-644cc665c9-5zmrp which contains the Nginx docker container.

However, had we set linkerd.io/inject: enabled at the namespace level, a long-running sidecar would be injected into the otherwise short-lived pod in (1). This long-running sidecar would prevent the pod as a whole from terminating naturally (by design a few seconds after creation) even if the original base admission container had terminated.

Without (1) being considered “done”, the creation of (2) would wait forever in an infinite timeout loop.

The above analysis only applies to that particular Helm chart. Other charts may have a different behaviour and different file structure for values.yaml. Be sure to check the nginx chart that you are using to set the annotation appropriately, if necessary.

Traefik

Traefik should be meshed with ingress mode enabled(*), i.e. with the linkerd.io/inject: ingress annotation rather than the default enabled. Instructions differ for 1.x and 2.x versions of Traefik.

Traefik 1.x

The simplest way to use Traefik 1.x as an ingress for Linkerd is to configure a Kubernetes Ingress resource with the ingress.kubernetes.io/custom-request-headers like this:

# apiVersion: networking.k8s.io/v1beta1 # for k8s < v1.19
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: emojivoto
  annotations:
    ingress.kubernetes.io/custom-request-headers: l5d-dst-override:web-svc.emojivoto.svc.cluster.local:80
spec:
  ingressClassName: traefik
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

The important annotation here is:

ingress.kubernetes.io/custom-request-headers: l5d-dst-override:web-svc.emojivoto.svc.cluster.local:80

Traefik will add a l5d-dst-override header to instruct Linkerd what service the request is destined for. You’ll want to include both the Kubernetes service FQDN (web-svc.emojivoto.svc.cluster.local) and the destination servicePort.

To test this, you’ll want to get the external IP address for your controller. If you installed Traefik via Helm, you can get that IP address by running:

kubectl get svc --all-namespaces \
  -l app=traefik \
  -o='custom-columns=EXTERNAL-IP:.status.loadBalancer.ingress[0].ip'

You can then use this IP with curl:

curl -H "Host: example.com" http://external-ip

Traefik 2.x

Traefik 2.x adds support for path based request routing with a Custom Resource Definition (CRD) called IngressRoute.

If you choose to use IngressRoute instead of the default Kubernetes Ingress resource, then you’ll also need to use the Traefik’s Middleware Custom Resource Definition to add the l5d-dst-override header.

The YAML below uses the Traefik CRDs to produce the same results for the emojivoto application, as described above.

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: l5d-header-middleware
  namespace: traefik
spec:
  headers:
    customRequestHeaders:
      l5d-dst-override: "web-svc.emojivoto.svc.cluster.local:80"
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
  creationTimestamp: null
  name: emojivoto-web-ingress-route
  namespace: emojivoto
spec:
  entryPoints: []
  routes:
  - kind: Rule
    match: PathPrefix(`/`)
    priority: 0
    middlewares:
    - name: l5d-header-middleware
    services:
    - kind: Service
      name: web-svc
      port: 80

GCE

The GCE ingress should be meshed with ingress mode enabled(*), i.e. with the linkerd.io/inject: ingress annotation rather than the default enabled.

This example shows how to use a Google Cloud Static External IP Address and TLS with a Google-managed certificate.

# apiVersion: networking.k8s.io/v1beta1 # for k8s < v1.19
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: emojivoto
  annotations:
    ingress.kubernetes.io/custom-request-headers: "l5d-dst-override: web-svc.emojivoto.svc.cluster.local:80"
    ingress.gcp.kubernetes.io/pre-shared-cert: "managed-cert-name"
    kubernetes.io/ingress.global-static-ip-name: "static-ip-name"
spec:
  ingressClassName: gce
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

To use this example definition, substitute managed-cert-name and static-ip-name with the short names defined in your project (n.b. use the name for the IP address, not the address itself).

The managed certificate will take about 30-60 minutes to provision, but the status of the ingress should be healthy within a few minutes. Once the managed certificate is provisioned, the ingress should be visible to the Internet.

Gloo

Gloo should be meshed with ingress mode enabled(*), i.e. with the linkerd.io/inject: ingress annotation rather than the default enabled.

As of Gloo v0.13.20, Gloo has native integration with Linkerd, so that the required Linkerd headers are added automatically. Assuming you installed Gloo to the default location, you can enable the native integration by running:

kubectl patch settings -n gloo-system default \
  -p '{"spec":{"linkerd":true}}' --type=merge

Gloo will now automatically add the l5d-dst-override header to every Kubernetes upstream.

Now simply add a route to the upstream, e.g.:

glooctl add route --path-prefix=/ --dest-name booksapp-webapp-7000

Contour

Contour should be meshed with ingress mode enabled(*), i.e. with the linkerd.io/inject: ingress annotation rather than the default enabled.

The following example uses the Contour getting started documentation to demonstrate how to set the required header manually.

Contour’s Envoy DaemonSet doesn’t auto-mount the service account token, which is required for the Linkerd proxy to do mTLS between pods. So first we need to install Contour uninjected, patch the DaemonSet with automountServiceAccountToken: true, and then inject it. Optionally you can create a dedicated service account to avoid using the default one.

# install Contour
kubectl apply -f https://projectcontour.io/quickstart/contour.yaml

# create a service account (optional)
kubectl apply -f - << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: envoy
  namespace: projectcontour
EOF

# add service account to envoy (optional)
kubectl patch daemonset envoy -n projectcontour --type json -p='[{"op": "add", "path": "/spec/template/spec/serviceAccount", "value": "envoy"}]'

# auto mount the service account token (required)
kubectl patch daemonset envoy -n projectcontour --type json -p='[{"op": "replace", "path": "/spec/template/spec/automountServiceAccountToken", "value": true}]'

# inject linkerd first into the DaemonSet
kubectl -n projectcontour get daemonset -oyaml | linkerd inject - | kubectl apply -f -

# inject linkerd into the Deployment
kubectl -n projectcontour get deployment -oyaml | linkerd inject - | kubectl apply -f -

Verify your Contour and Envoy installation has a running Linkerd sidecar.

Next we’ll deploy a demo service:

linkerd inject https://projectcontour.io/examples/kuard.yaml | kubectl apply -f -

To route external traffic to your service you’ll need to provide a HTTPProxy:

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: kuard
  namespace: default
spec:
  routes:
  - requestHeadersPolicy:
      set:
      - name: l5d-dst-override
        value: kuard.default.svc.cluster.local:80
    services:
    - name: kuard
      port: 80
  virtualhost:
    fqdn: 127.0.0.1.nip.io

Notice the l5d-dst-override header is explicitly set to the target service.

Finally, you can test your working service mesh:

kubectl port-forward svc/envoy -n projectcontour 3200:80
http://127.0.0.1.nip.io:3200

Kong

Kong should be meshed with ingress mode enabled(*), i.e. with the linkerd.io/inject: ingress annotation rather than the default enabled.

This example will use the following elements:

• The Kong chart
• The emojivoto example application

Before installing emojivoto, install Linkerd and Kong on your cluster. When injecting the Kong deployment, use the --ingress flag (or annotation).

We need to declare KongPlugin (a Kong CRD) and Ingress resources as well.

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: set-l5d-header
  namespace: emojivoto
plugin: request-transformer
config:
  remove:
    headers:
    - l5d-dst-override # Prevents open relay
  add:
    headers:
    - l5d-dst-override:$(headers.host).svc.cluster.local
---
# apiVersion: networking.k8s.io/v1beta1 # for k8s < v1.19
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: emojivoto
  annotations:
    konghq.com/plugins: set-l5d-header
spec:
  ingressClassName: kong
  rules:
  - http:
      paths:
      - path: /api/vote
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              name: http
      - path: /api/list
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              name: http

Here we are explicitly setting the l5d-dst-override in the KongPlugin. Using templates as values, we can use the host header from requests and set the l5d-dst-override value based off that.

Finally, install emojivoto so that it’s deploy/vote-bot targets the ingress and includes a host header value for the web-svc.emojivoto service.

Before applying the injected emojivoto application, make the following changes to the vote-bot Deployment:

env:
# Target the Kong ingress instead of the Emojivoto web service
- name: WEB_HOST
  value: kong-proxy.kong:80
# Override the host header on requests so that it can be used to set the l5d-dst-override header
- name: HOST_OVERRIDE
  value: web-svc.emojivoto

Haproxy

Haproxy should be meshed with ingress mode enabled(*), i.e. with the linkerd.io/inject: ingress annotation rather than the default enabled.

The simplest way to use Haproxy as an ingress for Linkerd is to configure a Kubernetes Ingress resource with the haproxy.org/request-set-header annotation like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: emojivoto
  annotations:
    kubernetes.io/ingress.class: haproxy
    haproxy.org/request-set-header: |
      l5d-dst-override web-svc.emojivoto.svc.cluster.local:80      
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

Unfortunately, there is currently no support to do this dynamically in a global config map by using the service name, namespace and port as variable. This also means, that you can’t combine more than one service ingress rule in an ingress manifest as each one needs their own haproxy.org/request-set-header annotation with hard coded value.

EnRoute OneStep

Meshing EnRoute with linkerd involves only setting one flag globally:

apiVersion: enroute.saaras.io/v1
kind: GlobalConfig
metadata:
  labels:
    app: web
  name: enable-linkerd
  namespace: default
spec:
  name: linkerd-global-config
  type: globalconfig_globals
  config: |
        {
          "linkerd_enabled": true
        }        

EnRoute can now be meshed by injecting Linkerd proxy in EnRoute pods. Using the linkerd utility, we can update the EnRoute deployment to inject Linkerd proxy.

kubectl get -n enroute-demo deploy -o yaml | linkerd inject - | kubectl apply -f -

The linkerd_enabled flag automatically sets l5d-dst-override header. The flag also delegates endpoint selection for routing to linkerd.

More details and customization can be found in, End to End encryption using EnRoute with Linkerd

Ingress details

In this section we cover how Linkerd interacts with ingress controllers in general.

In general, Linkerd can be used with any ingress controller. In order for Linkerd to properly apply features such as route-based metrics and traffic splitting, Linkerd needs the IP/port of the Kubernetes Service. However, by default, many ingresses do their own endpoint selection and pass the IP/port of the destination Pod, rather than the Service as a whole.

Thus, combining an ingress with Linkerd takes one of two forms:

1. Configure the ingress to pass the IP and port of the Service as the destination, i.e. to skip its own endpoint selection. (E.g. see Nginx above.)

2. If this is not possible, then configure the ingress to pass the Service IP/port in a header such as l5d-dst-override, Host, or :authority, and configure Linkerd in ingress mode. In this mode, it will read from one of those headers instead.

The most common approach in form #2 is to use the explicit l5d-dst-override header.