Monitor Kubernetes Apps through Metrics, Logs and Traces (MLT)
Metrics, Logs and Traces (MLT) are the three pillars of monitoring that can give us complete observability of a system.
In this article, we’re going to set up all three of MLT on a Kubernetes platform to monitor an app made with microservices.
What are metrics, logs and traces?
Metrics
Metrics show the measure of how a system resource is used. They are typically shown in numbers. For example-
- How much CPU was used in the past hour?
- How much disk space is consumed?
- How much bandwidth has been used?
Logs
Logs are events that a running software records. For example-
- Log the stack trace of a runtime error
- Log when a user accesses the system
Traces
Traces show the path of a program’s execution. In a modern distributed system, a request from a client may get processed through several services. It is important to know through which path a request got processed and how much time was needed in each node so that errors and bottlenecks can be identified.
Technologies required
Implementing all of this requires a bunch of different technologies. It’s easy to get lost, so take it easy. Let’s introduce the programs we’re gonna use.
- Grafana [Dashboard]: This is a central dashboard where we will observe every MLT data collected by different services
- Prometheus [Monitoring]: This collects kubernetes/pods/containers metrics (CPU, Memory, Bandwidth, etc) and sends them to grafana
- Promtail [Logging]: This collects log data from containers
- Loki [Logging]: This aggregates and stores all logs collected by Promtail and forwards them to Grafana
- Opentelemetry [Tracing]: This is used to instrument an app to collect traces and then forward them to Jaeger
- Jaeger [Tracing]: This collects tracing information for viewing. It can also optionally send them to grafana
The following figure summarizes our implementation.
Installation
We usually make deployments in Kubernetes through YAML files containing our preferred deployment. However, writing such deployment files can be difficult and tedious when you’re not a systems expert. Fortunately, there is Helm which is sort of a package manager for Kubernetes. With helm, services can be installed in a Kubernetes cluster with a few commands.
In this tutorial, we will be using helm to deploy services in Kubernetes. Also, make sure you have kubectl installed and configured to manage your cluster.
Install kubectl and configure
$ sudo snap install kubectl --classic
## This example is for AWS
$ aws eks --region eu-west-1 update-kubeconfig --name name-of-cluster
## For GCP and Azure check out their CLI config documentation
Install Helm
$ sudo snap install helm --classic
Or try another method from their official docs
Install the MLT Services
The procedures in this section have been made by following these official docs:
We will deploy the required services in the monitor namespace. So let’s create this namespace first.
$ kubectl create namespace monitor
Now, install the services. Make sure you have sufficient resources in the cluster before you proceed.
$ helm repo add grafana https://grafana.github.io/helm-charts
$ helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
$ helm repo update
## Copy the values file from below
$ helm upgrade --install loki grafana/loki-stack --namespace monitor --values values.yaml
$ helm upgrade --install jaeger jaegertracing/jaeger
Put the below config values in a values.yaml before running the grafana helm chart.
grafana:
image:
repository: grafana/grafana
tag: 8.0.6
enabled: true
persistence:
enabled: true
size: 500Mi
# existingClaim: loki-grafana
prometheus:
enabled: true
alertmanager:
persistentVolume:
enabled: false
server:
persistentVolume:
enabled: false
loki:
enabled: true
image:
repository: grafana/loki
tag: 2.2.1
persistence:
enabled: true
size: 3Gi
# existingClaim: loki-grafana
promtail:
image:
repository: grafana/promtail
tag: 2.2.1
More config variables can be found at the following:
Configuring the Services
Accessing the Grafana Dashboard
Access the grafana dashboard by making a proxy.
kubectl -n monitor port-forward service/loki-grafana 3000:80
Now hit on localhost:3000 and you should see the grafana dashboard.
Use, username: admin, and password as the output of the following command.
$ kubectl get secret --namespace monitor loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Setup prometheus metrics dashboard
Grafana has tons of awesome dashboards built by the open source community. We will proceed to use those ones. To import a graph go to sidebar > create > import.
In the “import via grafana.com” input box enter ids of community-built dashboard. Search grafana.com for the dashboards. Or you can use mine below favorites.
Best kubernetes cluster dashboards for grafana:
- Summary dashboards
- Kubernetes Cluster: 7249
- K8s Cluster Summary: 8685
- Kubernetes / Node Exporter Full: 12132
- Specific dashboards
- Kubernetes / Compute Resources / Namespace (Pods): 12117
- Kubernetes / Compute Resources / Node (Pods): 12119
- Kubernetes / Compute Resources / Pod: 12120
- Kubernetes / Networking / Cluster: 12124
- Kubernetes / Networking / Namespace (Pods): 12125
Here’s an example of what you could see.
Setup loki logs dashboard
On the left-hand side panel, go to create > dashboard. Click on add an empty panel. In this panel
- On the top right, change visualization to logs
- In data source, select loki
- Click on “Log browser” and select labels with which you’d like to filter the logs. I personally use namespaces and pods most.
Save the dashboard and you should see something like this!
Do check out the live tail feature, where the logs are live tailed as they are generated from the pods. Very useful for debugging applications.
Legacy config
# Install prometheus
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus prometheus-community/prometheus
kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-np
# Install grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana --set persistence.enabled=true,persistence.type=pvc,persistence.size=10Gi
kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-np
# Use the password provided by this to login
kubectl get secret --namespace default grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
# Install loki
helm repo add loki https://grafana.github.io/loki/charts
helm repo update
helm upgrade --install loki loki/loki-stack --set grafana.enabled=true
helm install loki-stack grafana/loki-stack \
--create-namespace \
--namespace loki-stack \
--set promtail.enabled=true,loki.enabled=true,loki.persistence.size=100Gi
Sharing is caring!