This is part 3 of our 5-part EKS security blog series. Don’t forget to check out our previous blog posts in the series:
- Part 1 — Guide to Designing EKS Clusters for Better Security
- Part 2 — Securing EKS Cluster Add-ons: Dashboard, Fargate, EC2 components, and more
Securing your Elastic Kubernetes Service (EKS) cluster’s network traffic and access is crucial for the entire cluster’s security and operation. Follow the below recommendations and best practices to protect your Kubernetes network on EKS.
Install Calico for Cluster Network Controls
Why: By default, network traffic in a Kubernetes cluster can flow freely between pods and also leave the cluster network altogether. In an EKS cluster, by extension, because pods share their node’s EC2 security groups, the pods can make any network connection that the nodes can, unless the user has customized the VPC CNI, as discussed in the Cluster Design blog post.
Creating restrictions to allow only necessary service-to-service and cluster egress connections decreases the number of potential targets for malicious or misconfigured pods and limits their ability to exploit the cluster resources.
The Calico CNI (Container Network Interface) offers the ability to control network traffic to and from the cluster’s pods by implementing the standard Kubernetes Network Policy API. Calico also offers some custom extensions to the standard policy type. Network policies can control both ingress and egress traffic. Different pod attributes, like labels or namespaces, can be used in addition to standard IP CIDR blocks to define the rules.
What to do: Install Calico and create network policies to limit pod traffic only to what is required. See our posts on guidelines for writing ingress and egress network policies. Test the policies to make sure they block unwanted traffic while allowing required traffic. Note that the NetworkPolicy resource type can exist in a cluster even though the cluster’s network setup does not actually support network policies. Therefore it would be possible to successfully create the policies, but they would have no effect.
Note that open-source Calico does not support Windows at this point, nor do most of the other CNI Network Policy providers. If you need Windows nodes, you may want to place them on dedicated VPC subnets and use VPC network ACLs to limit traffic to and from the nodes. Limiting pod-to-pod traffic is not possible without a cluster-aware solution, though.
Limit Network Access to the Kubernetes API Endpoint
Why: By default, EKS leaves the Kubernetes API endpoint, the management interface to the control plane, fully open to the Internet. However, EKS runs the API server with the --anonymous-auth=true flag, which allows unauthenticated connections and which EKS does not allow the user to disable. Even if anonymous users are not granted any Kubernetes RBAC privileges, this option still poses a danger. We have already seen at least one major security bug around allowing anonymous connections, last year’s Billion Laughs Attack. Because of the potential for vulnerabilities like that bug and in general for the Kubernetes API server, the endpoint should be protected by limiting network access to the API service only to trusted IP addresses.
What to do: EKS provides a couple options for protecting a cluster’s API endpoint.
- Disable the public endpoint and only use a private endpoint in the cluster’s VPC. This method provides the best protection but access to the endpoint from outside the VPC, if needed, would require going through a bastion host.
- Restrict the IP addresses that can connect to the public endpoint by using an allow list of CIDR blocks.
In addition to protecting the API service from Internet traffic, use network policies to block traffic from pods in the cluster to the API endpoint, only allowing workloads which require access.
Enable the Private Endpoint for the Kubernetes API
Why: As mentioned above, by default, all EKS clusters have only an endpoint publicly available on the Internet. As a side effect, traffic between the API server and the nodes in the cluster’s VPC leaves the VPC private network.
What to do: Enable the private endpoint for your cluster. EKS supports having both a public and private endpoint enabled for a cluster, so even if you still require the public endpoint, you can still keep your cluster traffic inside the VPC by using a private endpoint.
Block Access to Kubelet
Why: The kubelet runs on every node to manage the lifecycle of the pod containers assigned to that node. The kubelet needs strong protection because of its tight integration with the node’s container runtime. Even though EKS runs the kubelet with anonymous authentication disabled and requires authorization from the TokenReview API on the cluster API server, blocking access to its port from the pod network provides additional safeguards for this critical service.
What to do: After installing the Calico CNI, create a GlobalNetworkPolicy to prevent all pods from connection to the kubelet’s port 10250/TCP.
apiVersion: crd.projectcalico.org/v1
kind: GlobalNetworkPolicy
metadata:
name: deny-kubelet-port
spec:
types:
- Egress
egress:
- action: Deny
protocol: TCP
destination:
nets:
- 0.0.0.0/0
ports:
- 10250
source: {}Secure Service Load Balancers
Why: By default, when creating a Kubernetes Service of type LoadBalancer in an EKS cluster, the cluster’s AWS controller creates an Internet-facing ELB with no firewall restrictions other than those of the subnet’s VPC network ACL.
What to do: If only sources inside the cluster’s VPC need to access the service’s endpoint, add the following annotation to the Kubernetes Service manifest to create an internal load balancer: service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
If the load balancer needs to be Internet-facing but should not be open to all IP addresses, you can add the field loadBalancerSourceRanges to the Service specification.