If you’re running Kubernetes 1.6 or later, you can optionally make use of Kubernetes’ new support for RBAC (role-based access control), which allows you to restrict who can access the Kubernetes API on the cluster and what they can do with it. However, when upgrading to an RBAC-enabled cluster you can run into issues, as many Kubernetes examples do not take into account the fact that certain API calls may be restricted.
In this post, we’ll show you how to use Linkerd, our open source service mesh for cloud-native applications, with RBAC-enabled Kubernetes clusters.
What is RBAC?
First, it’s helpful to understand what RBAC actually does. RBAC works by
defining a role that describes a set of permissions, and by then assigning
that role to relevant users/service accounts. In Kubernetes RBAC, these roles
restrict which Kubernetes verbs can be used (e.g.
which Kubernetes resources they can be applied to (e.g.
for example, we can create a
Role (called, for example, “read-only”) that only
watch on pod resources. And we can then create
to assign this “read-only” role to whichever “subjects” need them, e.g. the
“qa-bot” service account.
In order for Linkerd to operate in an RBAC-enabled cluster, we need to make sure that the types of access that Linkerd needs to the Kubernetes APIs are allowed. Below, we’ll walk through how to do this. If you just want the completed config, you can skip to the bottom—or just use linkerd-rbac-beta.yml (stored in our linkerd-examples repo).
We’ll be setting up the permission by creating a
ClusterRole and a
ClusterRoleBinding, illustrated below.
Granting Linkerd access to an RBAC Kubernetes Cluster
When used with a Kubernetes cluster, Linkerd uses its
io.l5d.k8s “namer” to do
service discovery against the Kubernetes API. (Of course, this namer can be used
in conjunction with other service discovery mechanisms, allowing Linkerd to
bridge Kubernetes and non-Kubernetes systems—but that’s a later blog post).
Linkerd only requires read access, and only needs access to access the
endpoints Kubernetes resources. We can capture this access via
the following Kubernetes config:
--- # grant linkerd/namerd permissions to enable service discovery kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: linkerd-endpoints-reader rules: - apiGroups: [""] # "" indicates the core API group resources: ["endpoints", "services", "pods"] # pod access is required for the *-legacy.yml examples in linkerd-examples verbs: ["get", "watch", "list"]
For simplicity’s sake, at this point we could just assign this role to the
default service account (which is the account Kubernetes assigns to you when
you create a
if you don’t specify one):
--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: linkerd-role-binding subjects: - kind: ServiceAccount name: default # change this to your service account if you’ve specified one namespace: default roleRef: kind: ClusterRole name: linkerd-endpoints-reader apiGroup: rbac.authorization.k8s.io
Linkerd now has the access it needs to function in a Kubernetes environment. In production, however, you might want to use a dedicated service account—see below.
If you’re using
Namerd as a
control plane to dynamically change routing configuration across all Linkerd
for why you might want to do this), you’ll need some additional permissions.
Namerd needs access to a Kubernetes
ThirdPartyResource to store its routing
rules (“dtabs”). In our example namerd.yml, we’ve added this resource as
d-tab.l5d.io. We can allow Namerd read and write access to this resource using
the following role:
--- # grant namerd permisisons to third party resources for dtab storage kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: namerd-dtab-storage rules: - apiGroups: ["l5d.io"] resources: ["dtabs"] verbs: ["get", "watch", "list", "update", "create"]
Similar to above, we’ll assign the role to the
default service account with a
--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: namerd-role-binding subjects: - kind: ServiceAccount name: default namespace: default roleRef: kind: ClusterRole name: namerd-dtab-storage apiGroup: rbac.authorization.k8s.io
Running Linkerd with a specified Service Account
In the previous sections, we used the default service account to run Linkerd.
For some use cases, however, you may want to create a dedicated service account
and assign the permissions to that account. You’ll also want to consider whether
you want roles to be cluster-scoped (
ClusterRoleBinding) or namespace-scoped
RoleBinding). Let’s go through how to configure permissions for a specific
linkerd-svc-account, starting from the
linkerd.yml config in linkerd-examples. We’ll add a
ServiceAccount config, and assign that service account to the pod. Here’s part
of the file, with
--- apiVersion: extensions/v1beta1 kind: DaemonSet metadata: labels: app: l5d name: l5d spec: template: metadata: labels: app: l5d spec: volumes: - name: l5d-config configMap: name: 'l5d-config' serviceAccount: linkerd-svc-account containers: - name: l5d image: buoyantio/linkerd:1.1.2 env: - name: POD_IP valueFrom: fieldRef: fieldPath: status.podIP args: - /io.buoyant/linkerd/config/config.yaml ports: - name: outgoing containerPort: 4140 hostPort: 4140 - name: incoming containerPort: 4141 - name: admin containerPort: 9990 volumeMounts: - name: 'l5d-config' mountPath: '/io.buoyant/linkerd/config' readOnly: true - name: kubectl image: buoyantio/kubectl:v1.4.0 args: - 'proxy' - '-p' - '8001' --- apiVersion: v1 kind: ServiceAccount metadata: name: linkerd-svc-account
Then we’ll change the subject in your linkerd-rbac-beta.yml to reference this new service account:
--- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: linkerd-role-binding subjects: - kind: ServiceAccount name: linkerd-svc-account namespace: default roleRef: kind: ClusterRole name: linkerd-endpoints-reader apiGroup: rbac.authorization.k8s.io
And that’s it! The Linkerd pods now use the
linkerd-svc-account and have the
Putting it all together
For a complete Kubernetes config file that uses all of the above, just use this file: linkerd-rbac.yml. This config will allow Linkerd and Namerd to have all the access needed to the Kubernetes API with the default service account. If you’d like to set this up using a dedicated service account, you’ll need to modify linkerd-rbac-beta.yml, as described in the previous section. We hope this post was useful. We’d love to get your thoughts. Please join us in the Linkerd Discourse forums and the Linkerd Slack channel! And for more walkthroughs of how to use Linkerd’s various features on Kubernetes, see our Service Mesh For Kubernetes blog series.