Due to internal policies it might be mandatory to only allow mTLS-protected traffic inside your cluster between your services. To break this down into steps that can be easily implemented we need:
- a way to enable mTLS in the cluster, for this we are going to use Linkerd
- an Ingress Controller, since Linkerd does not provide one by default, the choice here is going to be the NGINX Ingress Controller
- an approach that lets us ensure that every new service (be it a Deployment, Statefulset, DaemonSet or ReplicaSet), gets automatically meshed and thus enables mTLS traffic to and from the aforementioned service, the final piece to our puzzle is going to be one of OPA Gatekeeper’s new mutating CRDs.
The installation of Linkerd is pretty straighforward and can easily be done by following the installation docs, so I won’t be going through it here.
NGINX Ingress Controller
The tricky part is meshing the Ingress Controller and the Ingress resources themselves. It is important to note that most Ingress Controllers select the endpoints themselves and do not pass the Services, just the corresponding Pod IPs and ports. This is also the case for the NGINX Ingress Controller: to avoid reloads on Endpoint changes, every time an endpoint changes the controller gets the endpoints from all the services and stores those endpoints in a shared memory zone and whenever a request comes in it picks one of the endpoints through a load balancing algorithm.
The problem with this approach is that to allow Linkerd to get route-based metrics and traffic splitting it needs the Service IP and port, not just the individual Pod IP and ports.
To skip this endpoint selection, you can just add the
nginx.ingress.kubernetes.io/service-upstream: "true" annotation to your Ingress resources.
To mesh the Controllers themselves you need to add the
linkerd.io/inject: enabled annotation to the Controller deployment and if you are deploying the NGINX Ingress Controller through a Helm chart then you have to add the annotation to the Deployment in the
values.yaml like this:
controller: podAnnotations: linkerd.io/inject: enabled
Just like the installation of Linkerd, this can also be done pretty easily following the docs of Gatekeeper, so we will focus on creating our CRD objects.
Note that the mutation CRD feature of Gatekeeper is currently (as of writing this) in beta, but I haven’t had any issues with them yet.
The CRD we are going to use to enforce automatic meshing of services is the
AssignMetadata CRD, which triggers the event of modifying the metadata section of the given resource upon its creation.
In our AssignMetadata object we add the
linkerd.io/inject: enabled annotation to every ReplicaSet, Deployment, DaemonSet and StatefulSet that is going to created in the given namespace, thus automatically meshing the services and enforcing mTLS between them.
apiVersion: mutations.gatekeeper.sh/v1beta1 kind: AssignMetadata metadata: name: nginx-annotation-assigner spec: match: scope: Namespaced kinds: - apiGroups: ["apps"] kinds: ["ReplicaSet", "Deployment", "DaemonSet", "StatefulSet"] location: 'metadata.annotations."linkerd.io/inject"' parameters: assign: value: "enabled"
- Linkerd docs: https://linkerd.io/2.11/getting-started/
- OPA Gatekeeper docs: https://open-policy-agent.github.io/gatekeeper/website/docs/
- NGINX Ingress Controller docs: https://kubernetes.github.io/ingress-nginx/