Beyond Patching: Fixing kubectl cp CVE-2019-11251

The Kubernetes project released patches yesterday for kubectl 1.13, 1.14, and 1.15, and also released kubectl 1.16.0 along with the release of Kubernetes 1.16. The previous versions were patched to address ongoing security vulnerabilities with the kubectl cp subcommand that could allow critical files to be overwritten or exfiltrated by accidental or malicious replacements when copying from a running container.

Fixing CVE-2019-11251

To address CVE-2019-11251, update all installations of the kubectl program to 1.13.11, 1.14.7, 1.15.4, or 1.16.0. Note that the fix in 1.16.0 results in different behavior than for the earlier minor releases. The three patch releases attempt to fix the problematic symbolic link handling while still permitting “safe” links. kubectl 1.16.0 removes support for symbolic links in the kubectl cp subcommand altogether.

Make sure all users of kubectl update any copies they may have on their laptops and that kubectl is also updated on any bastion (jump host in a secure DMZ network) or shared hosts or container images. It would be better still to find workarounds that avoid using kubectl cp or doing direct file transfers to and from production containers.

While applying security fixes is always important, keep reading for reasons why patching alone in the case of kubectl cp may not be enough to safeguard your clusters and Kubernetes API users.

The Current Issue

The kubectl cp subcommand allows users to copy files or recursive directories from a container in a Kubernetes pod to their laptop or vice versa. However, because Linux filesystems, as with most standard UNIX-like operating systems, support symbolic links (“symlinks”), what can look like an ordinary file can actually be a reference to a real file (or directory or another symlink, etc.). Symlinks can be very useful for many valid use cases, but if they are not dereferenced carefully in secure contexts, they can pose a security risk. A critical file, like /etc/passwd or a binary run as root, could possibly be overwritten, or sensitive files can be extracted to a remote system, unless care is taken to evaluate which files will be copied and where they actually reside in the filesystem.

Issues in kubectl cp around the handling of symlinks have already been the subject of CVE-2019-1002101 and CVE-2019-11246, but the patches for those previous vulnerabilities did not completely fix the issue, leading to CVE-2019-11251. The risk addressed in the patch for CVE-2019-11251 seems to be limited to using kubectl cp to copy from a container to the client system, much like the previous two CVEs.

The Ongoing Issue

The following reasons help explain why kubectl cp continues to pose a security risk even after two previous patches.

• kubectl cp is a wrapper for the kubectl exec command, which uses the exec subresource of the core Kubernetes Pod API.
• On the container side, kubectl cp relies on the tar command in the container’s default execution PATH.
  • tar (an acronym for “tape archive”) has a 40-year history, beginning as a tool to back up to and restore UNIX filesystems from tape. As such, by default, it tries to capture everything it finds to retain backup integrity. This completeness is at odds with selectively choosing “safe” files to read and write, as is required in a running production container.
  • Linux distributions typically ship with GNU tar, which unfortunately has its own long history of vulnerabilities.
  • kubectl on the client side is responsible for all checking and handling of the safety and correctness of the archive contents, whether they are being copied to or from the container. It doesn’t examine the container filesystem directly.
  • A malicious process or simple container build error could put another executable named tar earlier in the execution PATH.
• Because the degree of security best practices to which the container was built lies outside both the control and, at least as far as Kubernetes is concerned, the responsibility of kubectl, relying on the container’s image and runtime integrity and on assumptions that the container will behave in a certain way leave the door open for further accidental side effects or new security vulnerabilities with kubectl cp.
• kubectl 1.16.0 “fixes” the vulnerability by removing all support for symlinks. Unfortunately, the release notes direct users that still need symlink functionality to use kubectl exec + tar, thereby recreating the exact runtime conditions that started this chain of CVEs, without even the partial protections the ensuing patches have added.

Protecting Your Clusters Beyond Patching

Given the continuing possible risks that come with using even an up-to-date version of kubectl cp, the best way to protect your clusters is to avoid the command. Enforcing the usage of up-to-date clients may be easier said than done in some situations, but you can still take strong steps to remove the risk.

• Set the security contexts of your pods to run as a non-root user, disallow privilege escalation, and mount the root filesystem read-only:

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    allowPrivilegeEscalation: false
    privileged: false
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 1000

• Use Pod Security Policies in your cluster to require the use of these security context settings.
• Do not grant “create” privileges on the core pod/exec API in Kubernetes RBAC Roles or ClusterRoles. Note that this step also disallows the kubectl exec command, which makes it an even better idea for security but may not be feasible in some environments’ workflows.
• You can try omitting the tar command from your container images, but this solution to prevent execution of kubectl cp against the running container is not guaranteed to be effective on its own, particularly if there are any writable paths in the container’s PATH or any shells or http clients installed in the container, which a malicious process may be able to use to craft its own replacement for tar.
• Best: in addition to these measures, find methods to avoid the need for direct file transfer from the container through the Kubernetes API and kubectl. For example, sidecar containers can be very helpful in pushing output files to a cloud bucket for collection.

The Future

For now, this ongoing reliance on the container itself to provide a reliable, predictable, and secure means to move files to and from running containers using the Kubernetes API will continue to carry risk. (Ephemeral containers, introduced in alpha in Kubernetes 1.16.0, may hold one key alternative in the future.) The best way to protect yourself now, in addition to keeping both your Kubernetes clusters and the tools that access them, like kubectl, updated, is to follow security best practices for your container images and Kubernetes clusters, including treating the deployed containers like immutable objects.