Monitor a MariaDB Replication Cluster on Kubernetes with Prometheus and Grafana

Introduction

MariaDB is an open source relational database management system that is widely used for enterprise applications and mission-critical use cases. One of its key features is data replication, which allows data to be mirrored across multiple nodes and thereby avoid critical data loss. To make it easy to deploy MariaDB in production Kubernetes environments, Bitnami offers a MariaDB Helm chart. The Bitnami MariaDB chart sets up a master-slave MariaDB streaming replication cluster on Kubernetes that is easily configurable, fault-tolerant and follows best practices for security and scalability.

However, for DBAs and administrators, setting up a MariaDB replication cluster on Kubernetes is only part of the challenge. Equally important for them is the ability to monitor the cluster on an ongoing basis so that they can proactively respond to cluster health events.

This guide covers the process of installing Prometheus and Grafana on Kubernetes and deploying a MariaDB replication cluster with Prometheus metrics enabled. This gives DBAs and system administrators a way to visually monitor MariaDB cluster health and replication activity, obtain up-to-the-minute metrics and gain better insight into cluster usage.

Assumptions and prerequisites

This guide makes the following assumptions:

Step 1: Install Prometheus

The first step is to install Prometheus in the cluster, as follows:

helm repo update
helm install --name prometheus stable/prometheus --version 8.14.* \
  --set ingress.enabled=false \
  --set server.service.type=ClusterIP

Prometheus does not implement an authentication mechanism. Therefore, for security and to prevent the service being accessible externally, Ingress is disabled and a cluster IP address is assigned.

Once the deployment is complete, note the DNS name for the service within the cluster. You will need this name when configuring Grafana in a later step.

Prometheus service name
Tip

If you wish to log in to the Prometheus dashboard, follow the post-deployment instructions to proxy HTTP connections using kubectl.

Step 2: Install Grafana

The next step is to install Grafana using Helm. There are various options available here, depending on your access requirements.

Access Grafana through the Kubernetes proxy

You can install Grafana such that the service is only available through the Kubernetes proxy. This is the most secure option, as it precludes any external access. Use the command below, replacing the GRAFANA-PASSWORD placeholder with an actual password:

helm install --name grafana stable/grafana --version 3.5.11 \
    --set adminPassword=GRAFANA-PASSWORD

Configure kubectl to forward requests to Grafana using the commands below:

export POD_NAME=$(kubectl get pods --namespace default -l "app=grafana,release=grafana" -o jsonpath="{.items[0].metadata.name}")
kubectl --namespace default port-forward $POD_NAME 3000

Once the deployment is complete, you should be able to access Grafana by browsing to http://localhost:3000 on the host system and logging in with the username admin and the configured password.

Access Grafana over HTTP using a domain name

Tip

If you wish to access Grafana externally through a domain name, you must first install the NGINX Ingress controller.

To configure Grafana to be accessible externally through a domain name, use the command below. Replace the GRAFANA-PASSWORD placeholder with an actual password and the DOMAIN placeholder with the domain name you wish to use.

helm install --name grafana stable/grafana --version 3.5.11 \
    --set adminPassword=GRAFANA-PASSWORD \
    --set ingress.enabled=true \
    --set ingress.hosts={DOMAIN}

Once the deployment is complete, you should be able to access Grafana by browsing to http://DOMAIN and logging in with the username admin and the configured password.

Access Grafana over HTTPS using a domain name

Tip

If you wish to access Grafana externally through a domain name and with TLS, you must first install the NGINX Ingress controller and configure cert-manager for certificate generation.

To configure Grafana to be accessible over HTTPS and with a domain name, use the commands below. Replace the GRAFANA-PASSWORD placeholder with an actual password and the DOMAIN placeholder with the domain name you wish to use.

cat >grafana-values.yaml <<EOF
adminPassword: GRAFANA-PASSWORD
ingress:
  enabled: true
  hosts:
    - DOMAIN
  tls:
    - hosts:
        - DOMAIN
      secretName: grafana-tls
  annotations:
    kubernetes.io/tls-acme: "true"
EOF

helm install --name grafana stable/grafana --version 3.5.11 -f grafana-values.yaml 

Once the deployment is complete, you should be able to access Grafana by browsing to https://DOMAIN and logging in with the username admin and the configured password.

Here is what you should see after logging in:

Grafana dashboard

Step 3: Create a Prometheus data source in Grafana

Follow these steps:

  • From the Grafana dashboard, click the "Add data source" button.
  • On the "Choose data source type" page, select "Prometheus".
Grafana data source
  • On the "Settings" page, set the URL for the Prometheus server to http://NAME, where NAME is the DNS name for the Prometheus service obtained at the end of Step 1. Leave all other values at their default.
Grafana data source configuration
  • Click "Save & Test" to save and test the configuration. If everything is configured correctly, you should see a success message like the one below.
Grafana test* Click the "Dashboards" tab and import the available Prometheus dashboards (optional).Grafana Prometheus dashboards

Step 4: Deploy the MariaDB replication cluster

Once Prometheus and Grafana are configured, you can proceed to deploy the MariaDB replication cluster using the Bitnami MariaDB Helm chart. With the chart, Bitnami provides two configuration files: values.yaml, which initializes the deployment using a set of default values and is intended for development or test environments, and values-production.yaml, which is intended for production environments.

Deploy the MariaDB cluster with one master and two slaves using the production configuration using the command below. Replace the MARIADB-ADMIN-PASSWORD and MARIADB-REPL-PASSWORD placeholders with the database administrator account and replication account password respectively. You can also optionally create a MariaDB user account for application use by specifying values for the USER-PASSWORD, USER-NAME and DB-NAME placeholders.

wget https://raw.githubusercontent.com/helm/charts/master/stable/mariadb/values-production.yaml      
helm install --name mariadb stable/mariadb -f values-production.yaml --version 6.5.6 \
    --set rootUser.password=MARIADB-ADMIN-PASSWORD \
    --set replication.password=MARIADB-REPL-PASSWORD \
    --set db.user=USER-NAME \
    --set db.password=USER-PASSWORD \
    --set db.name=DB-NAME \
    --set slave.replicas=1 \
    --set metrics.enabled=true

Note the metrics.enabled parameter, which enables the Prometheus exporter for MySQL server metrics.

Once the deployment is complete, note the instructions to connect to the database service and proceed to the next step.

Step 5: Add MariaDB dashboards to Grafana

You can now proceed to add dashboards to Grafana for MariaDB monitoring. A number of community-created dashboards are available on the Grafana website and in the Percona GitHub repository.

Here is an example of adding the Percona MySQL Overview dashboard to Grafana:

Grafana dashboard import
  • Click "Load" to load the data and then "Import" to import the dashboard.

The new dashboard should appear in Grafana.

To test the dashboard, connect to the MariaDB service using the instructions provided at the time of deploying the Helm chart and run a few example queries. You should see your activity reflected in the MySQL Overview chart, as in the example below:

Grafana MySQL dashboard

You now have a MariaDB cluster running on Kubernetes with Prometheus metrics enabled and available through Grafana.

Appendix: Scale the deployment

You can easily scale the cluster up or down by adding or removing nodes. For example, to scale the number of slaves up to 5, get the name of the slave StatefulSet and then use kubectl, as shown below:

kubectl get sts -l "release=mariadb,component=slave"
kubectl scale sts/mariadb-slave --replicas=5

Useful links

To learn more about the topics discussed in this guide, use the links below: