Post

Meet Grafana LOKI, a Log Aggregation System for Everything

I’ve been on a quest to find a new logging system.I’ve use quite a few in the past, some open source, some proprietary, and some home grown, but recently I’ve decided to switch.I’ve switched to Grafana Loki for all of my logs for all of my systems - this includes machines, devices, docker systems and hosts, and my all of my kubernetes clusters.If you’re thinking of using Grafana and are also looking for a fast way to log all of your systems, join me as we discuss and configure Grafana Loki.

📺 Watch Video

Don’t want to host it yourself? Check out Grafana Cloud and sign up for a free account https://l.technotim.live/grafana-labs

Docker Setup

See this post on how to install docker and docker-compose

Running the container

If you’re using Docker compose

1
2
3
4
5
6
7
mkdir grafana
mkdir loki
mkdir promtail
touch docker-compose.yml
nano docker-compose.yml # copy the contents from below
ls
docker-compose up -d --force-recreate # be sure you've created promtail-config.yml and loki-config.yml before running this

docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
version: "3"
networks:
  loki:
services:
  loki:
    image: grafana/loki:2.4.0
    volumes:
      - /home/serveradmin/docker_volumes/loki:/etc/loki
    ports:
      - "3100:3100"
    restart: unless-stopped
    command: -config.file=/etc/loki/loki-config.yml
    networks:
      - loki
  promtail:
    image: grafana/promtail:2.4.0
    volumes:
      - /var/log:/var/log
      - /home/serveradmin/docker_volumes/promtail:/etc/promtail
    # ports:
    #   - "1514:1514" # this is only needed if you are going to send syslogs
    restart: unless-stopped
    command: -config.file=/etc/promtail/promtail-config.yml
    networks:
      - loki
  grafana:
    image: grafana/grafana:latest
    user: "1000"
    volumes:
    - /home/serveradmin/docker_volumes/grafana:/var/lib/grafana
    ports:
      - "3000:3000"
    restart: unless-stopped
    networks:
      - loki

Loki Config

1
nano loki/loki-config.yml

loki-config.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Promtail Config

1
nano promtail/promtail-config.yml

promtail-config.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

# local machine logs

- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
  
## docker logs

#- job_name: docker 
#  pipeline_stages:
#    - docker: {}
#  static_configs:
#    - labels:
#        job: docker
#        __path__: /var/lib/docker/containers/*/*-json.log

# syslog target

#- job_name: syslog
#  syslog:
#    listen_address: 0.0.0.0:1514 # make sure you also expose this port on the container
#    idle_timeout: 60s
#    label_structured_data: yes
#    labels:
#      job: "syslog"
#  relabel_configs:
#    - source_labels: ['__syslog_message_hostname']
#      target_label: 'host'

Loki Docker Driver

Install docker plugin

1
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

Edit docker daemon config

1
sudo nano /etc/docker/daemon.json

daemon.json

1
2
3
4
5
6
7
{
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "http://localhost:3100/loki/api/v1/push",
        "loki-batch-size": "400"
    }
}

Restart docker daemon.

1
 sudo systemctl restart docker

You will also need to recreate your containers after applying this setting *

LogQL sample queries

Query all logs from the varlogs stream

1
{job="varlogs"} 

Query all logs from the varlogs stream and filter on docker

1
2
{job="varlogs"}  |= "docker"

Query all logs from the container_name label of uptime-kuma and filter on host of juno

1
2
{container_name="uptime-kuma", host="juno"}

Read more about LogQL here

ARM CPU (Raspberry Pi)

There is a workaround for using this with ARM CPUs. Credit to AndreiTelteu for finding this in this discussion

delete /etc/docker/daemon.json

Add the vector service to the docker-compose.yml file

1
2
3
4
5
6
7
8
9
10
11
    vector:
        image: timberio/vector:0.18.1-debian
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - /home/serveradmin/docker_volumes/vector/vector-config.toml:/etc/vector/vector.toml:ro
        ports:
            - "8383:8383"
        restart: unless-stopped
        networks:
            - loki

Run this command

1
2
3
mkdir vector
cd vector
nano vector-config.toml

paste this config in the file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
[sources.docker-local]
  type = "docker_logs"
  docker_host = "/var/run/docker.sock"
  exclude_containers = []
 
  # Identify zero-width space as first line of a multiline block.
  multiline.condition_pattern = '^\x{200B}' # required
  multiline.mode = "halt_before" # required
  multiline.start_pattern = '^\x{200B}' # required
  multiline.timeout_ms = 1000 # required, milliseconds
 
[sinks.loki]
  # General
  type = "loki" # required
  inputs = ["docker*"] # required
  endpoint = "http://loki:3100" # required
  
  # Auth
  auth.strategy = "bearer" # required
  auth.token = "none" # required
  
  # Encoding
  encoding.codec = "json" # required
  
  # Healthcheck
  healthcheck.enabled = false # optional, default
  
  # Loki Labels
  labels.forwarder = 'vector'
  labels.host = ''
  labels.container_name = ''
  labels.compose_service = ''
  labels.compose_project = ''
  labels.source = ''
  labels.category = 'dockerlogs'

Credits to this post for the config file: grafana/loki#2361 (comment)

Kubernetes Setup

If you’re looking to set this up in kubernetes, see this post

⚙️ See all the hardware I recommend at https://l.technotim.live/gear

🚀 Don’t forget to check out the 🚀Launchpad repo with all of the quick start source files

This post is licensed under CC BY 4.0 by the author.