Dominik Táskai

Your average engineer

mTLS enforcement with OPA Gatekeeper

Published June 12, 2022

Due to internal policies it might be mandatory to only allow mTLS-protected traffic inside your cluster between your services. To break this down into steps that can be easily implemented we need:

  • a way to enable mTLS in the cluster, for this we are going to use Linkerd
  • an Ingress Controller, since Linkerd does not provide one by default, the choice here is going to be the NGINX Ingress Controller
  • an approach that lets us ensure that every new service (be it a Deployment, Statefulset, DaemonSet or ReplicaSet), gets automatically meshed and thus enables mTLS traffic to and from the aforementioned service, the final piece to our puzzle is going to be one of OPA Gatekeeper’s new mutating CRDs.

Linkerd

The installation of Linkerd is pretty straighforward and can easily be done by following the installation docs, so I won’t be going through it here.

NGINX Ingress Controller

The tricky part is meshing the Ingress Controller and the Ingress resources themselves. It is important to note that most Ingress Controllers select the endpoints themselves and do not pass the Services, just the corresponding Pod IPs and ports. This is also the case for the NGINX Ingress Controller: to avoid reloads on Endpoint changes, every time an endpoint changes the controller gets the endpoints from all the services and stores those endpoints in a shared memory zone and whenever a request comes in it picks one of the endpoints through a load balancing algorithm.

The problem with this approach is that to allow Linkerd to get route-based metrics and traffic splitting it needs the Service IP and port, not just the individual Pod IP and ports.

To skip this endpoint selection, you can just add the nginx.ingress.kubernetes.io/service-upstream: "true" annotation to your Ingress resources.

To mesh the Controllers themselves you need to add the linkerd.io/inject: enabled annotation to the Controller deployment and if you are deploying the NGINX Ingress Controller through a Helm chart then you have to add the annotation to the Deployment in the values.yaml like this:

controller:
  podAnnotations:
    linkerd.io/inject: enabled

OPA Gatekeeper

Just like the installation of Linkerd, this can also be done pretty easily following the docs of Gatekeeper, so we will focus on creating our CRD objects.

Note that the mutation CRD feature of Gatekeeper is currently (as of writing this) in beta, but I haven’t had any issues with them yet.

The CRD we are going to use to enforce automatic meshing of services is the AssignMetadata CRD, which triggers the event of modifying the metadata section of the given resource upon its creation.

In our AssignMetadata object we add the linkerd.io/inject: enabled annotation to every ReplicaSet, Deployment, DaemonSet and StatefulSet that is going to created in the given namespace, thus automatically meshing the services and enforcing mTLS between them.

apiVersion: mutations.gatekeeper.sh/v1beta1
kind: AssignMetadata
metadata:
  name: nginx-annotation-assigner
spec:
  match:
    scope: Namespaced
    kinds:
    - apiGroups: ["apps"]
      kinds: ["ReplicaSet", "Deployment", "DaemonSet", "StatefulSet"]
  location: 'metadata.annotations."linkerd.io/inject"'
  parameters:
    assign:
      value: "enabled"

Sources: