The Kubernetes team has released patches for the recently disclosed “Billion Laughs” vulnerability, that allowed an attacker to perform a Denial-of-Service (DoS) attack on the Kubernetes API server by uploading a maliciously crafted YAML file. With those patches comes the disclosure that the vulnerability was more severe than previously announced, as it could even be triggered by unauthenticated users (in Kubernetes 1.13) or any authenticated user, even when only granted read access via RBAC (Kubernetes 1.14+).
What you need to do now: upgrade your Kubernetes clusters
Before diving into the specifics, we strongly suggest that you upgrade your Kubernetes clusters to a recent version including a patch for the vulnerability: 1.16.2, 1.15.5, 1.14.8, or 1.13.12. Note that Kubernetes versions 1.12 and older are no longer supported, and since they do not receive patches for newly discovered vulnerabilities, running your clusters at such an old version puts your workloads and data at extreme risk. We also strongly recommend adopting best practices for securing access to the Kubernetes API of your clusters to reduce future risk.
Extended scope of the vulnerability
The original vulnerability could be triggered by uploading a maliciously crafted YAML file to the Kubernetes API server, that despite its small size could cause the server to exhaust its CPU and memory resources, thus becoming unresponsive.
Sending a YAML file to the server (the example used a ConfigMap, but any other resource type works as well) requires an attacker to possess the privilege of performing HTTP POST, PUT, or PATCH requests, corresponding to the create, edit, and patch verbs in Kubernetes RBAC configurations. Usually, unauthenticated users, or users that are merely authenticated but not granted any other RBAC permissions, do not possess these capabilities. Since RBAC constraints are evaluated before parsing the YAML payload (the operation to be performed as well as the affected resource types are derived from the request URL and the request method, not from the payload itself), merely performing an HTTP POST request with a malicious payload has no adverse effect if the request is denied by RBAC.
There is however one notable exception: the default system:basic-user role (which as of Kubernetes 1.13 is automatically granted to every user, including anonymous; since Kubernetes 1.14+ it is still automatically granted to every authenticated user) allows creation of SelfSubjectAccessReviews and SelfSubjectRuleReviews in the authorization.k8s.io API group. These are used to allow a user to check whether they could perform a certain operation based on their RBAC privileges, without actually performing it. The kubectl auth can-i subcommand is implemented this way (e.g., kubectl auth can-i create deployment -n myns tests if the active user is allowed to create a deployment in the myns namespace). Even though creating these resources is intended to be side-effect free (they do not get persisted), the ability to perform a create action on any single resource type is sufficient to exploit the vulnerability.
Please keep in mind that merely deleting the system:basic-user cluster role or the corresponding binding is not sufficient, as they are part of the Kubernetes RBAC bootstrap resources, and will be recreated/reset upon the restart of a Kubernetes API server. To remove the privileges of anonymous users in a persistent manner, follow the mitigation steps outlined at https://issue.k8s.io/83253 or or simply run the following command:
kubectl auth reconcile --remove-extra-subjects --remove-extra-permissions -f 'https://github.com/kubernetes/kubernetes/files/3735508/rbac.yaml.txt'Note that this can at best be considered a stopgap measure, we strongly recommend to instead upgrade to one of the patched versions.
Protecting yourself against future vulnerabilities
Aside from the urgent request to always run recent versions, we once more feel the need to reiterate the recommendation that access to the Kubernetes API server should be locked down:
- Ensure the API server is only accessible from trusted subnets (e.g., via appropriate firewall rules). Even better, do not expose it on the public internet at all, but only in a VPC network.
- Ensure that anonymous authentication is disabled.
Unfortunately, most cloud providers not only expose Kubernetes API servers on the public internet by default, but also keep anonymous authentication enabled in their managed Kubernetes offering.
Follow the appropriate link below for instructions on controlling network access to the Kubernetes API for your clusters. If your provider is not on the list, check their documentation.