Run Elastic Stack with Prometheus Metrics on Kubernetes
Introduction
The Elastic Stack evolved from the ELK Stack. "ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana.
- Elasticsearch is an open-source search and analytics engine.
- Logstash takes care of receiving, processing and transferring data to Elasticsearch.
- Kibana lets users visualize data with charts and graphs in Elasticsearch.
This guide walks you through the process of deploying an Elasticsearch cluster on Kubernetes. Bitnami's Elasticsearch Helm chart makes this a quick and error-free process. The best part about the Bitnami Helm chart is that it uses curated and trusted Bitnami images, thus making it very secure. Additionally, it provides production-ready default values, enabling you to deploy Elasticsearch with a single command instead of having a multi-step deployment run-book.
This guide also discusses deploying Elasticsearch with Prometheus metrics exporters, which can be further integrated with Tanzu Observability Service to configure dashboards. By relying on Kubernetes for the Elasticsearch cluster infrastructure, this approach avoids a single point of failure and makes it easier to scale out the Elasticsearch cluster as more computing resources become necessary.
In Elasticsearch 5.0, Ingest nodes were introduced as a way to process documents in Elasticsearch prior to indexing. They allow simple architectures with minimum components, where applications send data directly to Elasticsearch for processing and indexing. The main difference between Ingest nodes and Logstash, though, is that Ingest nodes are not able to pull data from an external source like a message queue or a database, while this is supported in Logstash. The choice of using Ingest nodes or Logstash therefore depends on the user's requirements.
Assumptions and prerequisites
This guide makes the following assumptions:
- You have a multi-node Kubernetes cluster (Kubernetes 1.12+) running with Helm (Helm 3.1.0) installed.
- You have the kubectl command line (kubectl CLI) installed and configured to work with your cluster.
- You have a Docker environment installed and configured. Learn more about installing Docker.
- You have PersistentVolume (PV) provisioner support in the underlying infrastructure and support for ReadWriteMany volumes for deployment scaling.
Deploy the Elastic Stack on Kubernetes
This guide uses VMware Tanzu Kubernetes Guest Cluster as the underlying infrastructure for the deployment of the Elastic Stack. The diagram below depicts the deployment architecture. Since this guide uses a three-node Elasticsearch cluster, it is recommended to have a three-node Kubernetes cluster for resiliency.
For a complete description of the important concepts and node types used in an Elasticsearch cluster, refer to the Elasticsearch reference guide.
Below is the list of Kubernetes objects that this chart deploys:
- Elasticsearch master Pods are deployed as a StatefulSet with a service to access them.
- Elasticsearch data Pods are deployed as a StatefulSet with a service to access them.
- Elasticsearch coordinating Pods are deployed with a service to access them.
- Ingest Pods are deployed with a service.
- PersistentVolumeClaims are created for the Elasticsearch data/master StatefulSet.
- Kibana Pods are deployed with a service.
Below are some important Helm chart parameters to keep in mind.
- global.kibanaEnabled: Use bundled Kibana
- master.replicas: Desired number of Elasticsearch master-eligible nodes
- ingest.enabled: Enable ingest nodes
See the complete list of parameters supported by the Bitnami Elasticsearch Helm chart.
Go ahead and deploy Elasticsearch on Kubernetes, as shown below:
# Add Bitnami Helm repository
$ helm repo add bitnami https://charts.bitnami.com/bitnami
# Update the repository
$ helm repo update
# Install the Elasticsearch chart with Kibana in the default namespace
$ helm install elastic bitnami/elasticsearch --set global.kibanaEnabled=true,master.replicas=3,ingest.enabled=true
Once the deployment is complete, check the deployed resources and access Kibana, as shown below:
# Check the PVCs created
$ kubectl get pvc
# Run the command to access Kibana
$ kubectl port-forward --namespace default svc/elastic-kibana 5601:5601
Elasticsearch is now up and running on the Kubernetes cluster. Access the Kibana Dashboard using the IP address to which the Kibana service was forwarded, as shown below:
Deploy the Elastic Stack on Kubernetes with Prometheus metrics
Prometheus (https://prometheus.io/) is a monitoring solution that has first class integration with Kubernetes. The Bitnami Elasticsearch Helm chart has the option of integrating with Prometheus via the Prometheus exporter that is distributed as part of the Bitnami Elasticsearch Docker image. This is turned off by default, but can be enabled via the metrics.enabled parameter.
Once you have enabled the Elasticsearch Prometheus metrics exporter, you can integrate with Tanzu Observability Service to configure dashboards in Wavefront. You can set up Wavefront using Bitnami Wavefront Helm chart which configures Wavefront Collectors as a DaemonSet, running one Collector pod on each node and a Wavefront Proxy Pod to feed real-time metrics into Tanzu Observability.
Alternatively you can set up an instance of Prometheus. The easiest way to stand one up is via the Bitnami Prometheus Operator Helm chart.
The diagram below depicts the Elasticsearch architecture with the Prometheus metrics exporter.
Apart from the important parameters mentioned in the earlier section, it is necessary to set metrics.enabled to true in order to enable the Prometheus metrics exporter when deploying the Helm chart.
# Install the chart with Elasticsearch Prometheus metrics exporter
$ helm install elastic bitnami/elasticsearch --set global.kibanaEnabled=true,master.replicas=3,ingest.enabled=true,metrics.enabled=true
Once the deployment is complete, check the deployed resources. In this example, the service endpoint is elastic-elasticsearch-metrics.
# Check for all the Kubernetes objects deployed
$ kubectl get all
# Check the Data and Master Nodes PVCs deployed
$ kubectl get pvc
# Validate if the metrics are being exported; as shown above the service elastic-elasticsearch-metrics can be used to fetch the metrics
$ kubectl exec elastic-elasticsearch-master-0 -it -- bash
$ curl elastic-elasticsearch-metrics:9114
You should see metrics being exported in Prometheus format. If you see this, Elasticsearch with Prometheus metrics is now up and running on the Kubernetes cluster!
Conclusion
As illustrated, the Bitnami Elasticsearch Helm chart makes deploying an Elasticsearch cluster along with observability a quick, easy and secure process with Bitnami's curated and trusted artifacts. This allows you to focus your time and efforts on business logic rather than deployment configuration using multi-step runbooks.