I have 5 servers used for the following purposes:
- edge1
- edge2
- services
- security
- media
Each server uses a Promtail docker container to push logs to a Loki instance hosted on the services server. All servers send docker logs and journal entries.
The promtail configuration looks like this:
server:
http_listen_address: 0.0.0.0
http_listen_port: 9080
positions:
filename: /etc/promtail/positions.yaml
clients:
- url: http://[services-ip]:3100/loki/api/v1/push
scrape_configs:
- job_name: edge1-docker-sock
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 15s
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'docker_container_name'
- source_labels: ['__meta_docker_container_log_stream']
target_label: 'docker_log_stream'
- source_labels: ['__meta_docker_container_label_logging_jobname']
target_label: 'docker_logging_job'
- source_labels: ['__meta_docker_network_ip']
target_label: 'docker_network_ip'
- source_labels: ['__meta_docker_network_name']
target_label: 'docker_network_name'
- job_name: edge1-journal
journal:
json: false
max_age: 12h
labels:
job: journal
host: edge1
relabel_configs:
- source_labels: ['__journal__systemd_unit']
target_label: journal_systemd_unit
- source_labels: ['__journal__systemd_user_unit']
target_label: journal_systemd_user_unit
- source_labels: ['__journal__hostname']
target_label: journal_hostname
- source_labels: ['__journal_syslog_identifier']
target_label: journal_syslog_identifier
- source_labels: ['__journal__transport']
target_label: journal_transport
- source_labels: ['__journal_priority_keyword']
target_label: journal_severity
The services server runs Loki in a container. This instance is added as a datasource to a Grafana where I can use the Explore function to validate my queries:
My crowdsec container is hosted on the same server as my traefik container, and the traefik logs folder is shared with the crowdsec container. I added specific Loki instances to my crowdsec aquisitions file for everything else:
source: file
filenames:
- /logs/web/*.log
log_level: warn
labels:
type: traefik
---
source: loki
url: http://192.168.1.93:3100/
query: |
{job="journal"}
limit: 1000
log_level: warn
labels:
type: syslog
---
source: loki
url: http://192.168.1.93:3100/
query: |
{docker_container_name="homeassistant"}
limit: 1000
log_level: warn
labels:
type: home-assistant
---
source: loki
url: http://192.168.1.93:3100/
query: |
{docker_container_name="immich_server"}
limit: 1000
log_level: warn
labels:
type: immich
---
source: loki
url: http://192.168.1.93:3100/
query: |
{docker_container_name="joplin-server"}
limit: 1000
log_level: warn
labels:
type: joplin-server
I added the various parsers to my crowdsec container:
# cscli parsers list
INFO Ignoring file /etc/crowdsec/parsers/s02-enrich/whitelists.yaml of type parsers
INFO Ignoring file /etc/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml of type parsers
PARSERS
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Name 📦 Status Version Local Path
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
crowdsecurity/appsec-logs ✔️ enabled 0.5 /etc/crowdsec/parsers/s01-parse/appsec-logs.yaml
crowdsecurity/cri-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s00-raw/cri-logs.yaml
crowdsecurity/dateparse-enrich ✔️ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
crowdsecurity/docker-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s00-raw/docker-logs.yaml
crowdsecurity/geoip-enrich ✔️ enabled 0.5 /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
crowdsecurity/home-assistant-logs ✔️ enabled 0.5 /etc/crowdsec/parsers/s01-parse/home-assistant-logs.yaml
crowdsecurity/http-logs ✔️ enabled 1.3 /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
crowdsecurity/sshd-logs ✔️ enabled 2.8 /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
crowdsecurity/syslog-logs ✔️ enabled 0.8 /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
crowdsecurity/traefik-logs ✔️ enabled 0.9 /etc/crowdsec/parsers/s01-parse/traefik-logs.yaml
crowdsecurity/whitelists 🏠 enabled,local /etc/crowdsec/parsers/s02-enrich/custom-whitelists.yaml
gauth-fr/immich-logs ✔️ enabled 0.2 /etc/crowdsec/parsers/s01-parse/immich-logs.yaml
instantdreams/local-addresses 🏠 enabled,local /etc/crowdsec/parsers/s02-enrich/local-addresses.yaml
instantdreams/zscaler-addresses 🏠 enabled,local /etc/crowdsec/parsers/s02-enrich/zscaler-addresses.yaml
xs539/joplin-server-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/joplin-server-logs.yaml
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
My issue is that the Loki parser doesn’t seem to separate by job:
# cscli metrics
Acquisition Metrics:
╭────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ file:/logs/web/access.log │ 23.51k │ 23.51k │ 1 │ 5.51k │ 41.29k │
│ loki:http://[services-ip]:3100/│ 23.98k │ - │ 23.98k │ - │ - │
╰────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯
How can I confirm that my docker logs for specific services are being parsed correctly, and that it isn’t just the last aquis.yaml file entry?
Is there a way to add additional labels to the aquisition entries such as job: immich and job: home-assistant?
Hopefully this is a useful use case that no one has yet delved into so we can document some examples.