Jan 20, 2021

Rancher Kubernetes Engine (RKE) Security Best Practice for Container and Runtime Security - Part 3 of 4

By: Michael Foster

This is the third installment in our four-part RKE security blog series. Don’t forget to follow along to our upcoming posts and get caught up on everything you missed!

Part 1 — Rancher Kubernetes Engine (RKE) Security Best Practices for Cluster Setup

Part 2 — Rancher Kubernetes Engine (RKE) Security Best Practices for Authentication, Authorization, and Cluster Access

Part 3 — Rancher Kubernetes Engine (RKE) Security Best Practice for Container and Runtime Security

Part 4 — Rancher Kubernetes Engine (RKE) Security Best Practice for Cluster Maintenance and Network Security

Container Image Security

Build Secure Images

Following best practices for building secure container images will minimize issues with running containers and simplify security updates. Use images that contain only the files required for the applications to run. Avoid using container images with frequently exploited or vulnerable tools like Linux package managers, web or other network clients, or Unix shells.

General Best Practices

  1. Use a minimal, secure base image.
  2. Only install tools needed for the container’s application. Debugging tools should be omitted from production containers.
  3. Remove exploitable tools as part of the initialization process with init-containers.
  4. Remove the package manager from the image as a Dockerfile build step.
  5. Keep images up-to-date. This practice includes watching for new versions of both the base image and any third-party tools you install.
  6. Maintain compatibility with proper tags. Avoid the use of the “latest” tag to help downstream users avoid compatibility issues.

Manage Images with a Registry

Rancher does not have an internally managed registry service, however, Docker allows a private registry to be set up. With private registries, organizations can manage the base images available to their teams by reviewing any images pulled from public registries. By using an internal registry, organizations can avoid third-party hosting and public image storing services, such as Docker Hub, and any outages or vulnerabilities associated with them.

Organizations can additionally implement policies for image use and vetting by taking advantage of a private image repository. Only after an image has passed the proper controls should it be allowed to be used to deploy containers into your cluster.

As mentioned in the cluster setup section, RKE allows a private registry to be defined in the cluster configuration files and provides for a default registry for image pulls to be set.

private_registries:
    - url: example_registry.com
     user: username
     password: password
     is_default: true #All system images are pulled using this registry

This will ensure that new clusters will not pull images from a third-party registry, such as Docker Hub, by accident.

Use an Image Scanner

Using containers free of known software security vulnerabilities requires ongoing vigilance. All the images deployed to a cluster should be scanned regularly by an image scanner that keeps an up-to-date database of Common Vulnerabilities and Exposures (CVEs). Several open-source and proprietary options exist, but be sure to choose a solution that scans vulnerabilities in operating system packages and third-party runtime libraries for your applications’ programming languages.

To address CVEs when they are found in your internally maintained images, your organization should have policies for updating and replacing images known to have serious, fixable vulnerabilities for images that are already deployed. Image scanning should be part of your CI/CD pipeline process, and images with high-severity, fixable CVEs should generate an alert and fail a build. We recommend considering open-source image scanning options and selecting a tool that would integrate with your image build pipeline.

Runtime Security

Namespaces

Kubernetes Namespaces provide scoping for cluster objects, allowing fine-grained cluster object management. Kubernetes Role-based Access Control (RBAC) rules for most resource types apply at the namespace level. Controls like Kubernetes Network Policies and many add-on tools and frameworks like service meshes are often scoped to the namespace level.

During the initial configuration of a cluster, focus on using one namespace per application to provide a greater opportunity to implement excellent security protocols. Since network policies and RBAC roles can be easily applied and finely tuned at the namespaces level, this enables more scalable security.

Avoid using the default namespace in any cluster outside of a development cluster. All applications require a specific namespace and should not be deployed into default due to ease of deployment.

Limit Container Runtime Privileges

Most containerized applications will not need any special host privileges on the node to function correctly. By following the principle of least privilege and minimizing your cluster’s running containers’ capabilities, you can significantly reduce the likelihood that a container is compromised.

Security Contexts

With Kubernetes, the PodSpec Security Context is used to define the exact runtime requirements for each workload. Similar to how RBAC resources control user access, administrators can use a pod- or container-level security context to manage permissions for containers. These controls can limit the resources, system calls, and filesystem access of the pods running in the cluster. If needed, the level of privileges is controlled for the application, giving them more permissive or more restrictive privileges.

RKE uses security contexts with the same functionality as outlined in the Kubernetes documentation. This includes:

  • Whether a pod or container has privileged access.
  • Discretionary Access Control such as the UID, GID of the container.
  • SELinux context of the container.
  • Configuration of allowable seccomp profiles.
  • configuration of allowable AppArmor profiles.
  • Use of host namespaces and networking.
  • Allocation of a fsGroup that owns the pod’s volumes.
  • Use of host directories as volumes.
  • Whether a container requires the use of a read-only root file system.
  • Usage of volume types.

These contexts can be applied at the pod or container level to allow flexibility around what each container requires.

Pod Security Policies (PSP)

When creating an RKE cluster, any default resources deployed in a namespace inherit a default security context associated with the namespace default service account through the use of Pod Security Policies. Rancher allows for two default policies on cluster creation; a restricted policy or an unrestricted policy. A PSP is a cluster-level resource that is applied through the use of RBAC controls. When creating a namespace for a specific application, ensure that the security context is set at minimum at the pod level and enforce a PSP for workloads in the namespace. A few rules to follow when creating and enforcing PSPs:

  1. Do not allow containers to run as root. Running as root creates the most significant risk since root access in a container is equal to root access on the underlying node.
  2. Do not use the host network or process space. Again, these settings create the potential for compromising the node and every container running on it.
  3. Do not allow privilege escalation.
  4. Use a read-only root filesystem for the container.
  5. Use the default (masked) /proc filesystem mount.
  6. Drop unused Linux capabilities and do not add optional capabilities that your application does not require. (Available capabilities depend on the container runtime in use on the nodes. GKE nodes can use either Docker or containerd, depending on the node image.)
  7. Use SELinux options for more fine-grained process controls.
  8. Give each application its own Kubernetes Service Account rather than sharing or using the namespace’s default service account.
  9. Do not mount the service account token in a container if it does not need to access the Kubernetes API.