Using kubecfg with Atlassian Bamboo to Deploy Kubernetes Workloads on GKE
Atlassian Bamboo is a continuous integration server that integrates with Atlassian Bitbucket, a web-based source code and on-premise solution for project management. It is used by a legion of developers in their daily jobs to improve their development speed and quality.
This guide will walk you through the process of setting up a complete development and deployment solution for Kubernetes running in Google Container Engine (GKE), using Atlassian Bamboo, Atlassian Bitbucket, and Kubecfg.
By following this guide, you will learn about:
- Jsonnet, a template language we can use to define complex data definitions with ease.
- Kubecfg, the tool that makes it possible to translate those templates into Kubernetes API objects, including creation, diffing, and deletion.
- How we can use them with Atlassian Bamboo to apply changes to our infrastructure, as defined by a GIT repository hosted by Atlassian Bitbucket.
- How to fetch the right credentials from Google to make Kubecfg talk with your cluster, using Kubernetes Secrets.
Kubecfg uses templates defined in Ksonnet or any other template library based on Jsonnet, and provides a simpler alternative to writing complex YAML files by hand for your Kubernetes configurations. Templates are stored in a source code repository and can be used to define complex environments, providing a flexible method for implementing Infrastructure as Code for Kubernetes.
The role of Kubecfg is to compile templates into YAML files which define Kubernetes API objects, and be able to create, diff, or delete those objects.
We will use Atlassian Bitbucket to host our Ksonnet definitions in a Git repository. We’ve previously published a sample repository using a Bitnami Nginx non-root image defined in Ksonnet. And this is the image we’ll deploy into our Kubernetes cluster.
Atlassian Bamboo will receive notifications of changes in the Bitbucket repository and carry out some actions as a result. In our demo, these actions will test the Ksonnet templates and deploy them to our Kubernetes cluster.
The goal of this “How to” is to provide guidance in order to implement a robust Infrastructure CI process on Kubernetes using Atlassian tools.
Assumptions and prerequisites
For this demo environment, we’ll deploy Bitbucket and Bamboo in the same GKE cluster where the non-root Nginx containers are going to be deployed, but we could host them elsewhere.
You can find the Dockerfiles and other Kubernetes resources for running Bitbucket and Bamboo in this repository.
This tutorial will take you through the following steps:
- Step 1: Deploying Bamboo and Bitbucket
- Step 2: Obtaining the “how to” source code and uploading it to Bitbucket
- Step 3: Defining a build plan in Bamboo
- Step 4: Defining a deployment project in Bamboo
- Step 5: Deploying the project to GKE
Step 1. Deploying Bamboo and Bitbucket
The first thing we need to do is to deploy Bamboo and Bitbucket in our Kubernetes cluster. You can find out how to do this in the repository README.md file.
An explanation of our Bamboo Dockerfile
Most of the magic will happen in the Bamboo Dockerfile provided in the repository. This section explains the main aspects of that Dockerfile, which will allow Bamboo to interact with our GKE Kubernetes cluster.
The Bamboo Dockerfile includes the following two lines, so Bamboo is able to execute the kubecfg command.
Kubecfg uses the same config file that kubectl uses. In a nutshell, it expects to find a ~/.kube/config file with the proper credentials.
To obtain the credentials to populate the ~/.kube/config file needed by kubecfg, we have created a Service Account, as explained in the README.md file. Once the secret is deployed, it will be presented to our Bamboo server as a Volume under the /var/lib/kubernetes/gcloud-svc-account directory, with the following files:
- gcloud-svc-account: the JSON file we obtained from creating the service account on the Google console.
- gcloud-cluster: your GKE cluster name.
- gcloud-cluster-zone: the availability zone where your cluster lives.
The secret will be used to create a ~/.kube/config file in the Bamboo container to be used by kubecfg. This way, you’re authorized to run kubectl/kubecfg commands. But to create the config file with the right contents, we need to install the gcloud suite of tools in our Bamboo server.
To do that, the Dockerfile entrypoint script does the following:
- Sets the gcloud tools directory
- Activates the gcloud service account using the file from the secret volume
- Creates the ~/.kube/config file which is going to be used by kubecfg
Now, the Bamboo server is ready to execute kubecfg commands against your cluster. To contact the Kubernetes cluster, a sidecar container handles the Kubecfg communication with the Kubernetes API server.
Step 2: Obtaining the “how to” source code and uploading it to Bitbucket
To obtain the source code, you will need to go to your Bitbucket server and add a new project called kubecfg-demo, and select “Import repositories from -> Git.” In the URL field, indicate https://github.com/bitnami/kubecfg-nonroot-nginx.git.
Once the repository is imported to your Bitbucket server, clone it to your workstation. You can get the URL by going to the web interface of Bitbucket and clicking “Actions -> Clone”. Then, simply copy the URL.
Step 3: Defining a build plan in Bamboo
In Atlassian speak, linking two applications means that you are sharing information and accessing one application’s functions and resources from within the other. By linking Bitbucket and Bamboo you’ll be able to launch a Bamboo build plan when a commit is done in the repository. Let’s create a build plan to build and test our Ksonnet templates on every repository commit.
Create the build plan
Click on “Create a build plan” in the upper Bamboo UI bar. A build plan contains all the steps you need to execute to build and test your software.
The project’s name is Kubecfg-demo, and the plan is named kubecfg-nginx, as shown below.
In the “Link repository to new build plan” field, we selected the “Bitbucket Server/Stash” option as repository host, selected “Bitbucket” as the server (thanks to it being a linked application), and the repository name that was cloned before.
Click the “Configure plan” button to define the steps for building and testing your software. The first one is created by Bamboo and is a task of type Source Code Checkout, which will fetch the source code from Bitbucket.
Compile Ksonnet definitions
The next step is to add another task of type script. This will execute the build.sh script you can find in the root directory of the kubecfg-nonroot-nginx repository, and its mission is to check whether the Ksonnet definition compiles successfully.
If you examine the build.sh script, you’ll see that it executes two commands:
kubecfg show nginx.jsonnet compiles the Ksonnet definitions files into a YAML file, ready to be consumed by Kubernetes.
kubecfg diff --diff-strategy subset nginx.jsonnet shows you the differences between the current state on the cluster and the changes that need to be made in the Kubernetes cluster to satisfy the state definition on the repository. This command is concatenated with an echo Unix command to exit with error code zero. This is because the diff command of kubecfg ends with an error code other than zero if it finds any difference between the desired state and the real state of the cluster. It’s very useful for detecting configuration drift (like manual changes) but in this case, we only want to show the differences.
Enable the build plan
The last step is to enable the build plan. Select the “Enable this plan?” checkbox, and then click the “Create” button. The first build will run.
Configure artifacts to be shared with Bamboo
To get started, click the “Action” gear icon and select “Configure Plan”.
To configure the artifacts, select the “Artifacts” tab and then click “Create definition” as shown below:
In the “Name” field, enter Ksonnet files and in the “Copy pattern” field, enter “*/” which indicates all files and subdirectories. Don’t forget to check the “Shared” checkbox. Your plan summary should look like this:
Great! Your code is ready to be deployed on Kubernetes. Let’s define a Deployment Project to do just that.
Step 4: Defining a deployment project in Bamboo
Create a deployment project
Once you’ve built your software , let’s deploy it with a deployment project.
Click the “Create” button in the top menu and select the “Create a deployment project” option. Next, you need to select the build plan you’re going to deploy, in this case, kubecfg-nginx:
You’ll see a message saying, “No environments found”. That’s because we’ve not defined where and how we want to deploy.
Create an environment
- Click on “Create a new environment”, and name it gcloud k8s cluster. Then, click on “Continue to task setup”.
As you can see, there are already two tasks defined: “Clean working directory” and “Artifact download”.
- In the “Artifact download” task, select the “Ksonnet files” option (the artifact definition we created a few seconds ago) and save the task.
Add another task, type script, and specify deploy.sh in the “script file” field.
Again, the deploy.sh script executes two kubecfg commands:
- kubecfg diff --diff-strategy subset nginx.jsonnet shows you the differences between the desired state of the cluster indicated in the Ksonnet files and the real cluster state
- kubecfg update nginx.jsonnet sends the compiled YAML files to the Kubernetes API server to be applied
When you click on “Finish the deployment plan”, you can see a summary of the project, where you can define advanced settings for the deployment such as release naming, project permissions, notifications, and automatic triggers, amongst other deployment options.
Step 5: Deploying the project to GKE
To deploy the project, go to the top menu in Bamboo, select “More -> Deploy -> All deployment projects” and click the “kubecfg-nginx-deployment” deploy icon.
Deploy a release
Select the first successful build plan, and set a release name. Click on the “Start deployment” button.
Now, go to your Kubernetes UI console, and you’ll able to see the new deployment with one Nginx pod running.
Cool! But this is only the beginning!
Modify your deployment definition
Let’s make a change on the Git repository you’ve cloned from Bitbucket to see how changes are propagated to your Kubernetes cluster.
Go to the repository in your disk, and edit the file nginx.jsonnet to change the value of replicas to 3, then commit and push the changes to Bitbucket Server.
Now for the magic: the push will trigger the Bamboo build plan that will compile the files into JSON, and will show you the differences against the actual cluster state in the logs:
Then go to the Deploy page and in the kubecfg-nginx-deployment, create a new release from the build result and name it nginx-3-pods. Next, deploy it.
Check the Kubernetes deployment pods
Head over to your Kubernetes UI again, and take a look at the number of pods in the nonroot-nginx deployment. You must see three pods now.
Managing Kubernetes workloads has never been so fun and easy!
- In this guide, you learnt a Kubernetes best practice used in production ready containers: avoid to running it as root user.
- Also, you learnt how to use Atlassian tools to implement a Continuous Integration workflow for workloads running on a Kubernetes cluster.
- Finally, you learnt how to by making changes in a repository that contains templates, Kubecfg can manage a complex infrastructure in a easy way.