kubernetessecure-kubernetes-services-with-ingress-tls-letsencrypt

Secure Kubernetes Services with Ingress, TLS and LetsEncrypt

Introduction

Kubernetes gives you a lot of flexibility in defining how we want our services to be exposed. You can configure your Service objects to ensure a group of pods are only accessible within the cluster, or enable access from outside the cluster. If your cloud provider of choice supports it, you can even request a load balanced IP address or hostname for your Service.

The Service object gives us a simple way to connect services running in pods, however if you find the need for setting up basic routing rules to different services or want to configure things like TLS, the Ingress resource provides a much more flexible way to configure external access.

Ingress can be backed by different implementations through the use of different Ingress Controllers. The most popular of these is the NGINX Ingress Controller, however there are other options available such as Traefik, Rancher, HAProxy, etc. Each controller should support a basic configuration, but can even expose other features (e.g. rewrite rules, auth modes) via annotations.

In this guide, we’ll take a look at setting up the NGINX Ingress Controller in our cluster and create Ingress routes for an example app. You’ll learn how Ingress objects are defined, including how to configure TLS and basic auth.

Prerequisites and Assumptions

This guide makes the following assumptions:

Step 1: Install the NGINX Ingress Controller

The first step is to install the NGINX Ingress Controller. The easiest way to get this running on any platform is using the stable Helm chart.

$ helm install stable/nginx-ingress --namespace kube-system

Note for kubeadm clusters, you should to enable hostNetwork and deploy a DaemonSet to ensure the Nginx server is reachable from all nodes.

$ helm install stable/nginx-ingress --namespace kube-system --set controller.hostNetwork=true,controller.kind=DaemonSet

Once the controller is created and running, we can access the NGINX server through the LoadBalancer IP or NodePort of its Service. For kubeadm, you can use the IP of any node.

External IP of Ingress controller

If everything is setup okay, you should get a “default backend - 404” response when accessing the Nginx server. This tells us that the controller doesn’t know where to route our request to - we haven’t configured any rules yet - so responds with the default backend.

Step 2: Deploy the example application

Now that we have an Ingress controller, we can go ahead and start creating some Ingress rules! First, though, we need an application to route to.

The Kubernetes charts repository has a lot of easily deployable apps on offer to test this out, this example uses the Joomla! chart to get an instance of the Joomla! CMS running in the cluster.

$ helm install stable/joomla --name ingress-example

By default, the chart will create a Service with a LoadBalancer IP. We can make sure Joomla! is working by visiting the external IP or NodePort in the browser. However, since we want to configure access to the CMS via an Ingress route, it would be better to only allow the Service to be exposed internally within the cluster (i.e. with ClusterIP type).

Change the Service to use the ClusterIP type and remove the associated node ports.

$ kubectl patch svc ingress-example-joomla --type='json' -p '[{"op":"remove","path":"/spec/ports/0/nodePort"},{"op":"remove","path":"/spec/ports/1/nodePort"},{"op":"replace","path":"/spec/type","value":"ClusterIP"}]'

Now we should no longer be able to access Joomla! through the external IP or node ports, but the Ingress controller will be able to proxy to it within the cluster.

Step 3: Create Ingress object

The Ingress definition is a straightforward list of routing rules, each describing the path and Service to proxy to. Since we are only serving a single application, we only need to define one rule.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: joomla-ingress
annotations:
    kubernetes.io/ingress.class: nginx
spec:
rules:
- http:
    paths:
    - path: /
        backend:
        serviceName: ingress-example-joomla
        servicePort: 80

Once created, the NGINX Ingress controller will pick up the new object and configure NGINX to serve the rules we created. For the above rule, an incoming request to the root path will reach the NGINX server which will then act as a reverse proxy to one of the available Joomla! pods. Because there is no hostname set for this rule, it will act as a wildcard. This means the external IP of the NGINX service and any hostname pointing at it will follow this rule.

Save the definition above into a file and create the object in your cluster.

$ kubectl apply -f basic-ingress.yaml

After a few seconds, Joomla! should be accessible via the NGINX service’s external IP (i.e. the same IP we used to reach the default backend in step 1). Note that the NGINX Ingress Controller forces a self-signed TLS certificate for wildcard routes, in the next step we will make this more secure by configuring a real TLS certificate for the Joomla! site.

Joomla! site behind a self-signed TLS certificate

Step 4: Configure TLS with LetsEncrypt and kube-lego

Now that we have re-enabled external access to our Joomla! instance, the next step is to give it a domain name and enable TLS. LetsEncrypt is a free TLS certificate authority, and using the kube-lego controller we can automatically request and renew LetsEncrypt certificates for public domain names, simply by adding a few lines to our Ingress definition!

In order for this to work correctly, a public domain name is required and should have an A record pointing to the external IP of the NGINX service. In these examples, we are using “YOUR_DOMAIN”, but these references should be replaced with a real domain name.

To complete this process, follow the steps below:

  • Install the kube-lego chart. In order to register with the LetsEncrypt service, an email must be provided when installing the chart (replace YOUR_EMAIL with a valid email address). By default, the chart uses the staging LetsEncrypt environment for testing, this can be changed by setting the production URL when installing the chart.

    $ helm install stable/kube-lego --namespace kube-system --set config.LEGO_EMAIL=YOUR_EMAIL,config.LEGO_URL=https://acme-v01.api.letsencrypt.org/directory
    
  • Set the hostname for Joomla! in the Ingress definition, and configure the TLS section. The tls-acme annotation is also added to tell the kube-lego controller to request a certificate for this Ingress rule. Replace “YOUR_DOMAIN” with the domain name configured to point to the NGINX service.

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: joomla-ingress
    annotations:
        kubernetes.io/ingress.class: nginx
        kubernetes.io/tls-acme: 'true'
    spec:
    rules:
    - host: YOUR_DOMAIN
        http:
        paths:
        - path: /
            backend:
            serviceName: ingress-example-joomla
            servicePort: 80
    tls:
    - secretName: joomla-tls-cert
        hosts:
        - YOUR_DOMAIN
    
  • Save the new definition to a file and apply the changes in the cluster.

    $ kubectl apply -f tls-ingress.yaml
    

kube-lego will pick up the change to the Ingress object, request the certificate from LetsEncrypt and store it in the “joomla-tls-cert” Secret. In turn, the NGINX Ingress Controller will read the TLS configuration and load the certificate from the Secret. Once the NGINX server is updated, a visit to the domain in the browser should present the Joomla! site over a secure TLS connection.

Joomla! site behind a secure TLS certificate

An alternative way to manage LetsEncrypt certificates on Kubernetes is using Kelsey Hightower’s kube-cert-manager. This controller makes use of ThirdPartyResources (now CustomResourceDefinitions) instead of Ingress to request certificates for domains. These certificates get stored in a Secret which you can then reference in the Ingress resource as above. This approach is more flexible as it doesn’t require the use of Ingress objects, you could consume the certificates directly in your applications, however kube-lego is simpler if you’re only using TLS in your Ingress.

Manually configuring a TLS certificate

LetsEncrypt is a great way to easily configure TLS for your services, but it’s also easy to use existing TLS certificates with Ingress objects. All you need to do is to create a Secret with the TLS certificate and reference it from the Ingress object (removing the tls-acme annotation if kube-lego is installed).

$ kubectl create secret tls custom-tls-cert --key /path/to/tls.key --cert /path/to/tls.crt

The above command will create the secret “custom-tls-cert” with the certificate, which can be referenced in the Ingress definition as follows.

...
tls:
- secretName: custom-tls-cert
    hosts:
    - YOUR_DOMAIN

Step 5: Setup a basic routing rule

So far, we have defined a simple route to just one application, however Ingress objects allow you to define multiple routes and paths under multiple hostnames. In addition to the Joomla! site, we want to setup a Ghost blog when visiting the “blog” subdomain.

To create an Ingress definition with multiple routes:

  • First, install the Ghost chart on the cluster. Make sure to replace YOUR_DOMAIN with your own domain. This time, we set the type of the Service when installing the chart so we don’t need to change it later.

    $ helm install stable/ghost --name ingress-blog --set ghostHost=blog.YOUR_DOMAIN,serviceType=ClusterIP
    
  • Now, update the Ingress definition to add the new routing rule for the Ghost blog on the subdomain. The subdomain must also be added to the TLS section. Remember to change YOUR_DOMAIN here too.

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: joomla-ingress
    annotations:
        kubernetes.io/ingress.class: nginx
        kubernetes.io/tls-acme: 'true'
    spec:
    rules:
    - host: YOUR_DOMAIN
        http:
        paths:
        - path: /
            backend:
            serviceName: ingress-example-joomla
            servicePort: 80
    - host: blog.YOUR_DOMAIN
        http:
        paths:
        - path: /
            backend:
            serviceName: ingress-blog-ghost
            servicePort: 80
    tls:
    - secretName: joomla-tls-cert
        hosts:
        - YOUR_DOMAIN
        - blog.YOUR_DOMAIN
    
  • Once again, save this to a file and apply the change.

    $ kubectl apply -f multi-app-ingress.yaml
    

With this change, the kube-lego controller will pick up the change to the TLS section and request a new certificate that covers both domains, and the NGINX Ingress controller will regenerate the NGINX configuration with the new virtual host for the “blog” subdomain. Once this configuration is updated, the Joomla! site will be accessible on the root domain and the Ghost blog can be reached on the “blog” subdomain.

Ghost blog on subdomain

Step 6: Configure basic auth and other Ingress options

The NGINX Ingress Controller exposes different options for configuring the NGINX server through annotations on the Ingress object. We’ll take a look at setting up HTTP Basic Auth as an example of one of these options:

  • First, we need to create the htpasswd file for storing the usernames and passwords. Create a Secret containing this file so the NGINX Ingress controller can use it.

    $ htpasswd -bc auth ingress-user ingress-password
    $ kubectl create secret generic joomla-basic-auth --from-file=auth
    
  • Now we need to add the “auth-type: basic” and “auth-secret: joomla-basic-auth” annotations to the Ingress definition. This tells the NGINX Ingress controller to configure basic auth for the virtual host and where to read the htpasswd file from. Remember to change “YOUR_DOMAIN”.

    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: joomla-ingress
    annotations:
        kubernetes.io/ingress.class: nginx
        kubernetes.io/tls-acme: 'true'
        ingress.kubernetes.io/auth-type: basic
        ingress.kubernetes.io/auth-secret: joomla-basic-auth
    spec:
    rules:
    - host: YOUR_DOMAIN
        http:
        paths:
        - path: /
            backend:
            serviceName: ingress-example-joomla
            servicePort: 80
    - host: blog.YOUR_DOMAIN
        http:
        paths:
        - path: /
            backend:
            serviceName: ingress-blog-ghost
            servicePort: 80
    tls:
    - secretName: joomla-tls-cert
        hosts:
        - YOUR_DOMAIN
        - blog.YOUR_DOMAIN
    
  • Save the above definition to a file and apply the changes.

    $ kubectl apply -f basic-auth-ingress.yaml
    

Once the NGINX Ingress controller has picked up the change and configured the NGINX service, a login prompt should appear when refreshing the Joomla! or Ghost site in your browser and logging in with “ingress-user” and “ingress-password” will grant you access.

Basic auth prompt

This guide walked through the Kubernetes Ingress object: what it is, how it’s different from a Service, how it’s configured. We looked at setting up a simple Ingress definition for our example Joomla! site, then extending it to secure with TLS encryption and adding a new rule to route to the Ghost blog. In the previous section we looked at configuring other options like Basic Auth for the NGINX Ingress controller. The documentation for the controller is a great resource for seeing what other options are available.