Running Helm in Production - Security Best Practices

Helm has become one the most popular package managers for Kubernetes. The goal of Helm is to help you manage Kubernetes applications using Charts. Helm charts are just "packages" that you can directly install in your Kubernetes cluster. They are really useful since they abstract all the complexity around ConfigMaps, Deployments, Volumes, etc. that otherwise you need to handle one by one, to deploy applications in Kubernetes.

When using Helm in production (i.e. in a Kubernetes cluster with security policies), it's necessary to understand how to properly set up this tool to avoid security issues. In this post, I will walk through some of the security challenges produced by Helm, explaining security best practices to avoid these issues and how Bitnami can help you overcome them with Kubeapps: An open source, web-based UI for deploying and managing applications in Kubernetes.

Installing Helm

Helm is very easy to install: once you get the helm CLI, you just execute a single command:

$ helm init
$HELM_HOME has been configured at /home/andres/.helm.

Tiller (the Helm server-side component) has been installed in your Kubernetes cluster.

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.
To prevent this, run `helm init` with the --tiller-tls-verify flag.
For more information on securing your installation, see:
Happy Helming!

So now you are ready to install your favorite applications in Kubernetes! But wait, something else should catch your attention:

Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.

That's indeed disturbing. Let's dig into that a little bit more.

Helm security challenges

Angus Lees, a Bitnami Kubernetes developer, wrote a very detailed article about what the security challenges of Helm are. The main issue is that Helm requires a server side named Tiller. This component is in charge of contacting the Kubernetes API in order to install, on behalf of the user, anything specified in a Helm chart.

This implies that Tiller:

  • will usually need admin privileges: If a user wants to install a chart that contains any cluster-wide element like a ClusterRole or a CustomResourceDefinition (CRD), Tiller should be privileged enough to create or delete those resources.
  • should be accessible to any authenticated user: Any valid user of the cluster may require access to install a chart.

That leads to a now-obvious security issue: escalation of privileges. Suddenly, users with minimum privileges are able to interact with the cluster as if they were administrators. The problem is bigger if a Kubernetes pod gets compromised: that compromised pod is also able to access the cluster as an administrator. That's indeed disturbing.

Mitigating the issues

The official Helm documentation explains a few hints to mitigate these problems. Unfortunately, they don't directly suit this case:

  • Reducing Tiller permissions may be an obvious option. That doesn't fit our needs though. You’ll probably want cluster administrators to be able to install charts with cluster-wide components.
  • Securing Tiller endpoint with a TLS certificate. With this certificate, users not only need to have a valid user, but they also need access to Tiller certificate to contact it. A compromised pod isn’t able to access Tiller since you should restrict access to the certificate by default. The problem is that managing access to the certificate is difficult to maintain. Now cluster administrators need to apply rules to allow or deny access for every new user.
  • [Running a Tiller instance per namespace. This way you can reduce Tiller permissions for certain instances, while leaving others privileged. Again, the downside with this solution is that it's difficult to maintain and now you are wasting resources, having duplicated deployments.

Tillerless and Helm v3

Helm maintainers are aware of the vulnerabilities that having a server side for Helm causes, so in the next major version, they'll get rid of Tiller. If you are curious about what's going to happen with this new version, find the design proposal here. Unfortunately, an official release date is not yet available, so for the time being, it is necessary to stick with the current version 2.

A special mention is deserved for what's called Tillerless Helm, which is about running Tiller in your local host rather than in the Kubernetes cluster. Tiller momentarily runs using the authentication information of the user who executes it, so it's not possible to escalate privileges.

Using this approach has a downside though. You still need to store information about the charts that you install in the cluster. That means that anyone using this solution will need to configure the namespace they are allowed to use, and they will need to run several commands (in different terminals) to deploy a chart. There is a plugin that does this for you, but in any case, this will divert from the default experience.

Using Kubeapps and Tiller-Proxy

Kubeapps is a web-based UI for managing applications in Kubernetes clusters. In other words, it allows you to discover and install Helm charts without the helm CLI, using a web interface.

Kubeapps deployment

Kubernetes Role-Based Access Control (RBAC) system backs Kubeapps. This means that to sign in, you need a Kubernetes API token, which is really easy to obtain. Learn how to do so in the getting started guide. Once users are logged in, they will be authenticated as specific Kubernetes users and they won't be able to escalate privileges. To achieve this, we have developed an authorization proxy that validates any action targeting Tiller. This simplified diagram explains how we do it:

tiller proxy diagram
  1. The user requests to install a chart (i.e. bitnami/wordpress).
  2. Before installing the chart, Kubeapps resolves the manifest with Tiller. This manifest contains all the resources required for the chart.
  3. For each one of the resources, Kubeapps checks that the authenticated user has permissions to create it in the given namespace. To do so, Kubeapps uses the Authorization API and is able to differ if the user has permissions to perform the requested action (i.e. create a Deployment in the namespace "default").
  4. Kubeapps installs the chart only if the request is valid.

With that set-up, it's really easy for users to install Helm charts without the security disadvantages of using a single Tiller for a Kubernetes cluster.

If you are interested in learning how to set up Helm, Tiller, and Kubeapps securely, check out our step-by-step guide.

Happy Helming!

This tutorial is part of the series

Best Practices for Creating Production-Ready Helm Charts

Learn how to create a custom Helm chart from scratch, the guidelines you need to follow to make production-ready charts, and which are the basic concepts you need to know for running Helm charts in production.