Post

Beautiful Dashboards with Grafana and Prometheus - Monitoring Kubernetes Tutorial

Grafana and Prometheus are a powerful monitoring solution.It allows you to visualize, query, and alert metrics no matter where they are stored.Today, we’ll install and configure Prometheus and Grafana in Kubernetes using kube-prometheus-stack. By the end of this tutorial you be able to observe and visualize your entire Kubernetes cluster with Grafana and Prometheus.

📺 Watch Video

A HUGE thanks to Datree for sponsoring this video!

Combat misconfigurations. Empower engineers.

https://www.datree.io

Getting Started

If you need to install a new kubernetes cluster you can use my Ansible Playbook to install one.

k3s

If you want to get metrics from your k3s servers, you will need to provide some additional flags to k3s.

Additional k3s flags used in the video:

1
extra_server_args: "--no-deploy servicelb --no-deploy traefik --kube-controller-manager-arg bind-address=0.0.0.0 --kube-proxy-arg metrics-bind-address=0.0.0.0 --kube-scheduler-arg bind-address=0.0.0.0 --etcd-expose-metrics true --kubelet-arg containerd=/run/k3s/containerd/containerd.sock"

helm

1
2
3
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Install helm

The helm chart we will be using to install Grafana, Preometheus, and Alert Manager is kube-prometheus-stack

Installing

Verify you can communicate with your cluster

1
kubectl get nodes
1
2
3
4
5
6
NAME     STATUS   ROLES                       AGE   VERSION
k3s-01   Ready    control-plane,etcd,master   10h   v1.23.4+k3s1
k3s-02   Ready    control-plane,etcd,master   10h   v1.23.4+k3s1
k3s-03   Ready    control-plane,etcd,master   10h   v1.23.4+k3s1
k3s-04   Ready    <none>                      10h   v1.23.4+k3s1
k3s-05   Ready    <none>                      10h   v1.23.4+k3s1

Verify helm is installed

1
helm version
1
version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}

Add helm repo

1
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Update repo

1
2
helm repo update

Create a Kubernetes Namespace

1
kubectl create namespace monitoring

Echo username and password to a file

1
2
echo -n 'adminuser' > ./admin-user # change your username
echo -n 'p@ssword!' > ./admin-password # change your password

Create a Kubernetes Secret

1
 kubectl create secret generic grafana-admin-credentials --from-file=./admin-user --from-file=admin-password -n monitoring

You should see

1
secret/grafana-admin-credentials created

Verify your secret

1
kubectl describe secret -n monitoring grafana-admin-credentials

You should see

1
2
3
4
5
6
7
8
9
10
11
Name:         grafana-admin-credentials
Namespace:    monitoring
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
admin-password:  9 bytes
admin-user:      9 bytes

Verify the username

1
kubectl get secret -n monitoring grafana-admin-credentials -o jsonpath="{.data.admin-user}" | base64 --decode

You should see

1
adminuser%

Verify password

1
kubectl get secret -n monitoring grafana-admin-credentials -o jsonpath="{.data.admin-password}" | base64 --decode
1
p@ssword!%

Remove username and password file from filesystem

1
rm admin-user && rm admin-password

Create a values file to hold our helm values

1
nano values.yaml

paste in values from here

Create our kube-prometheus-stack

1
helm install -n monitoring prometheus prometheus-community/kube-prometheus-stack -f values.yaml

Port Forwarding Grafana UI

(be sure to change the pod name to one that matches yours)

1
kubectl port-forward -n monitoring grafana-fcc55c57f-fhjfr 52222:3000

Visit Grafana

http://localhost:52222

If you make changes to your values.yaml you can deploy these changes by running

1
helm upgrade -n monitoring prometheus prometheus-community/kube-prometheus-stack -f values.yaml

Examples:

Traefik Ingress example

⚙️ See all the hardware I recommend at https://l.technotim.live/gear

🚀 Don’t forget to check out the 🚀Launchpad repo with all of the quick start source files

This post is licensed under CC BY 4.0 by the author.