The Kubernetes project recently completed a security audit, including a review of source code, system design, and live behavior in a test environment. The Security Audit Working Group released the results publicly, including:
- A final report detailing the audit process, describing the issues uncovered, and recommending technical or design changes to Kubernetes developers, and
- A threat model describing key Kubernetes components and how effectively they implement important security controls
The Working Group also released materials to help cluster administrators, operators, or developers apply better security practices to their Kubernetes clusters and applications:
- A white paper explaining important aspects of Kubernetes’ design and offering recommendations for operators and developers, and
- A threat-focused guide to securing or attacking Kubernetes clusters
What do I need to know?
We read through all 241 pages of results to identify the key takeaways — there’s a lot of good content, but here’s the tl;dr:
- The Kubernetes project and the Cloud Native Computing Foundation (CNCF) are making significant investments in product security.
- The security audit identified a number of security issues in specific components of Kubernetes, with severities ranging from Informational to High, and recommended improvements in certain areas. Kubernetes is handling the reported security problems responsibly, and you can already see some of the improvements.
- Configuring Kubernetes is complex, and it can be hard to keep track if your organization is shipping new code quickly. To be secure, you need to take steps to protect your infrastructure and your applications.
1. Kubernetes and CNCF are investing in product security
As more and more organizations adopt Kubernetes in production, we’ve seen increased focus on its security. Every large software system will have bugs, and Kubernetes is no exception. An important part of establishing trust is handling security problems openly, transparently, and effectively — and getting in the habit of doing so. In addition to the audit, the Kubernetes community has been openly documenting and using solid security practices; you may have seen these workflows in action in recent security update releases, like 1.15.3 for industry-wide HTTP/2 denial of service issues or 1.12.3 for a critical Kubernetes API security flaw.
You can’t just handle security problems as they come up, though — you have to go looking for them too. The CNCF, which hosts the Kubernetes project, has sponsored audits of a number of other projects recently, including CoreDNS, Envoy, and Prometheus, and now Kubernetes, and plans to conduct more.
Compared to a typical security audit of a commercial product, this one has been unusually open. The Working Group published details of its RFP selection process and various notes and working outputs. The Trail of Bits team even published its working observations and meeting notes, including comments from Kubernetes Special Interest Groups (SIGs) and others knowledgeable about how Kubernetes was built. And the Product Security Committee decided that all of the findings could be addressed publicly.
2. A number of security issues are being addressed
The audit team identified a number of specific security issues in Kubernetes — from secret tokens leaking into logs to minor mistakes in source code. The issues affect a variety of components and some only apply under certain configurations. For example, some problems only affect you if you use specific storage drivers or other plugins, or if you enable debugging logs.
The assessment also analyzes Kubernetes at a system level, exploring system design choices and the state of certain security-critical user-accessible features.
The executive summary of the final report explains the team’s view of the challenges facing Kubernetes:
Overall, Kubernetes is a large system with significant operational complexity. The assessment team found configuration and deployment of Kubernetes to be non-trivial, with certain components having confusing default settings, missing operational controls, and implicitly defined security controls. Also, the state of the Kubernetes codebase has significant room for improvement. The codebase is large and complex, with large sections of code containing minimal documentation and numerous dependencies, including systems external to Kubernetes. There are many cases of logic re-implementation within the codebase which could be centralized into supporting libraries to reduce complexity, facilitate easier patching, and reduce the burden of documentation across disparate areas of the codebase.
Kubernetes Security Assessment, Trail of Bits, p. 5
Many of the findings were treated as feature requests or would require long-term coordination because of Kubernetes’ compatibility and stability policies. In the meantime, users can implement configurations that compensate for some of the underlying issues.
We’ll highlight a few categories of findings here:
Problems in deprecated features: Early on, some Kubernetes features weren’t built as securely as they could have been. Some of these features live on even though more secure alternatives are available. For example, the audit recommends documenting that Secure Shell (SSH) Tunneling is deprecated and insecure (finding TOB-K8S-012), and that the HTTP Basic Authentication option, which stores passwords unencrypted in clear text, is for development purposes only (finding ATR-K8S-002).
Parsing problems: To do its job, Kubernetes has to accept user-provided configurations. However, some of the code handling this input could overflow or run out of memory (findings TOB-K8S-015, TOB-K8S-019, and TOB-K8S-020).
Information leaks: Environment details and secrets could leak under certain configurations: into logs, if a high-verbosity log level was set (finding TOB-K8S-001, now fixed); to clients through an accidental DNS zone transfer from CoreDNS (finding TOB-K8S-032); or to users with host access, via environment variables used for secrets (finding TOB-K8S-005).
Feature design issues: Resourceful adversaries can abuse certain features because of how they’re designed. For example, one finding described how users with limited access could find out more information about the cluster using readiness and liveness health-check probes, which are important operational features (finding TOB-K8S-024).
Feature requests: The audit identifies areas for improvement, like improving support for seccomp profiles, which limit what actions an individual application can take (finding TOB-K8S-002). This is a long-standing feature request dating back at least a few years. Similarly, a GitHub issue for the ability to revoke certificates was filed in 2015 (finding TOB-K8S-028).
Documentation: The audit recommends documentation improvements in some cases, for example to help users select the most secure configurations for data encryption. The community has also addressed some of the findings by updating documentation. For example, the Kubernetes documentation was edited to note that PodSecurityPolicy-based restrictions on host-path mounts don’t work if your application uses a PersistentVolumeClaim to claim the host path (a high-severity finding, TOB-K8S-038).
You can dig into the rest of the issues in the final report. Kubernetes developers have begun tackling the issues. To see the current status of all the remediations, check issue #81146 on the Kubernetes issue tracker on GitHub.
3. What you should do as a Kubernetes user
Many of the assessment findings will be more useful to developers working on the Kubernetes project than to end users. If you develop, operate, or secure applications running on Kubernetes, don’t let the details of the internal security improvements distract you from the audit’s important advice about how you can be more secure.
As in any infrastructure, the responsibility for the security of a Kubernetes cluster is shared. While Kubernetes itself is an important part of your security posture, so are your nodes, cloud provider or cluster manager (if applicable), container images, workload configurations, and deployment practices.
Many of the recommendations center on how to securely configure individual pods because of their important role in your overall security posture:
Often, compromising a Kubernetes cluster begins with first compromising a lower privileged Pod. The secure configuration of Pods is an often-overlooked aspect of the system.
Attacking Kubernetes, Atredis Partners, p. 46
Since pod configurations are so critical, you need to carefully audit how your applications are deployed and set limits on who can directly create them. Previous Kubernetes security issues like the subpath flaw (CVE-2017-1002101) have been especially dangerous if untrusted users have control over certain aspects of pod configuration.
We’ve covered many Kubernetes security controls in previous blog posts — including Kubernetes network policies, admission controllers, and how to securely build production-ready clusters — in posts we’ve contributed to the Kubernetes and CNCF blogs, and in a CNCF webinar. The audit highlights a number of potential security issues and steps you can take to be more secure. For each, we’ll include the content directly from the reports so you can find any additional context.
Use a read-only root file system
[The] root file system is not commonly read-only, allowing for additional tools to be installed.
Attacking Kubernetes, Atredis Partners, p. 46
Marking your container’s root file system read-only can be a powerful mitigation against attempts to gain a foothold in your infrastructure: the adversary’s job is more difficult if they can’t easily download new tools or programs.
Minimize image contents
File systems also often contain bash or package managers that further enable an attacker to gain a shell and install additional tools. An ideal installation should remove all non- essential binaries and prevent modification to the binaries that are required.
Attacking Kubernetes, Atredis Partners, p. 46
You can remove package managers and other useful tools from final images to make it harder for adversaries to abuse them. This practice can be difficult to implement in base images, since downstream builds may need these tools; however, you can use multi-stage builds or clear out unnecessary tools as a final build step to put your applications on a more secure footing.
Address application vulnerabilities
For cluster administrators, care should be taken that vulnerable applications and Pods are patched as soon as possible, so that Internal Attackers may not gain an initial foothold within the cluster.
Kubernetes Security Whitepaper, Trail of Bits, pp. 14-15
Managing vulnerabilities is an important part of securing your workloads. Container images help you do this effectively, since images can be scanned before they’re deployed, and they have digest identifiers that can be referenced between systems. Fixable vulnerabilities should be solved or mitigated as soon as possible as part of your team’s regular update workflow.
Pay attention to your Kubernetes RBAC configuration
All containers in a Pod run with a service account. If a service account is not provided, the default account is used. … Attack scenarios have been documented against third-party services which will orchestrate Pod deployment using overly permissive service accounts. In these instances, a compromise of a Pod container is catastrophic.
The level of access and permissions granted to the default service account vary greatly across deployments. In practice, they will likely be given some permissions that can be abused; however, they may not yield much access when using role-based access control (RBAC) authorization controls.
Attacking Kubernetes, Atredis Partners, p. 23
At the time of this report, Kubernetes mounted default credentials in every Pod; an Internal Attacker could use these credentials to access other resources within the cluster, such as the kublet (sic). From there, the Internal Attacker may be able to move laterally throughout the cluster to wider access.
Kubernetes Security Whitepaper, Trail of Bits, pp. 14-15
The Kubernetes API is a critical attack surface — it controls what deploys in your cluster, along with security-critical configurations. If access is too broad, an adversary could create unwanted containers in your cluster, modify important configurations, or otherwise abuse your infrastructure. Or, someone with good intentions might delete or modify resources across the entire cluster by mistake.
The Kubernetes RBAC feature is powerful but can be difficult to configure properly:
[Objects] can be composed by referencing objects that may not yet exist. Additionally, objects can be created even if the component using the object does not exist. This functionality can be very dangerous when constructing RBAC policies, since functionality must be tested to ensure the configuration works in the expected manner. This could lead an administrator to believe that policies are in effect, when in fact they are not.
Kubernetes Security Whitepaper, Trail of Bits, p. 16
And of course the legacy Attribute-Based Access Control (ABAC) controller should be disabled so that RBAC has an effect. ABAC has been disabled for a number of releases but it might still be enabled in your clusters if you upgraded from an older version.
Use namespaces
Kubernetes namespaces were developed as a method to help provide workload isolation. Running multiple, potentially multi-tenant, workloads in the same namespace sidesteps the protections of namespaces, resulting in a single large and flat namespace.
Kubernetes Security Whitepaper, Trail of Bits, p. 9
If you don’t explicitly put your applications in a separate Kubernetes namespace, they’ll end up in the default namespace and you’ll miss out on the natural boundaries created by namespaces. Some of these boundaries are important for security: for example, pods can mount secrets only from within their namespace, and Kubernetes RBAC authorization is defined along namespace boundaries.
Use Kubernetes network policies
Finally, ensure the container network interface is as restrictive as possible through the definition of cluster network policies.
Kubernetes Security Whitepaper, Trail of Bits, p. 17
Network policies let you say which applications can talk to each other and how. But, until you define Kubernetes network policies, each of your pods can talk to any other pod by default. Once you define your declarative network policies, you can test them just like you’d test any other code or configuration change, giving you robust, portable network segmentation.
To learn more about Kubernetes network policies, see our detailed Security Guide.
Get past default settings
Kubernetes contains many default settings which negatively impact the security posture of a cluster. These settings also have conflicting usage semantics, where some use either opt-in or opt-out specifications. The conflicting usage generally boils down to the preservation of backwards compatibility for both workload and component configurations. Ensuring appropriate configuration of all options requires significant attention by cluster administrators and operators.
Kubernetes Security Whitepaper, Trail of Bits, p. 17
Many of the Kubernetes default settings are great for getting started, but you should reconsider your privileges, configurations, and policies to minimize risk. Remember, network policies are strictly opt-in — until you set them, your network is wide open. Similarly, pods run as root and receive a Kubernetes API token by default. You can change all of these behaviors, but you have to take action to do so.
Secure your underlying infrastructure
While Kubernetes facilitates high-availability workload deployments, the underlying hosts, components, and environment of a Kubernetes cluster must be configured and managed. This management has a direct impact on the capabilities of the cluster, and affects the behavior of an operator’s composed objects.
Kubernetes Security Whitepaper, Trail of Bits, p. 5
Of course, it’s important to secure the infrastructure on top of which your containers run. The Center for Internet Security (CIS) benchmarks are a great place to start, but ongoing monitoring and configuration assessment are also critical. The audit whitepaper recommends certain additional practices (pp. 16-18):
- Restrict access to nodes and configure them securely; for example, make file permissions as restrictive as possible
- Segment the host network
- Secure the surrounding environment
- Implement logging and alerting
Where to go from here
If you use Kubernetes, you know how it can help you simplify repetitive operational tasks and accelerate business value. But you have to take certain steps to operate Kubernetes securely. Adopt the recommendations in the assessment and you’ll be well on your way to improved security. As the report notes:
Despite the results of the assessment and the operational complexity of the underlying cluster components, Kubernetes streamlines difficult tasks related to maintaining and operating cluster workloads such as deployments, replication, and storage management. Additionally, Kubernetes takes steps to help cluster administrators harden and secure their clusters through features such as Role Based Access Controls (RBAC) and various policies which extend the RBAC controls. Continued development of these security features, and further refinement of best practices and sane defaults will lead the Kubernetes project towards a secure-by-default configuration.
Kubernetes Security Assessment, Trail of Bits, p. 6
The StackRox Kubernetes Security Platform is purpose-built to secure your workloads and your Kubernetes and container infrastructure, helping you build more secure container images, deploy with hardened configurations, and monitor for security events at runtime. StackRox analyzes all the critical aspects of your Kubernetes and container configuration as part of our risk management, compliance, and configuration management features.
Use out-of-the-box or custom policies to enforce a read-only root file system, limit what tools are allowed in the images you deploy, identify unnecessary Kubernetes RBAC exposures, and more. Take advantage of our multi-factor risk profiling, which ties together vulnerabilities and other declarative data from images and deployments alongside runtime activity.
See all your deployments, dynamically ranked by risk, so you know what to fix first. And build better security in using our Kubernetes-native enforcement — we can automatically build updated network segmentation policies based on your application’s real-world traffic, and we use Kubernetes to enforce full-lifecycle security policies using admission control and other techniques.