Integrate CrowdSec metrics into Grafana Cloud

1

Hi there!

I’ve managed to integrate CrowdSec metrics into Grafana Cloud, so I’d like to share :smiley_cat:.

First, I’ve set up a … Grafana Cloud :wink: instance. For now I’m running the Pro trial, but I’m pretty confident I’ll could keep running on the free plan.
For your information, what’s included:

image
image1018×779 79 KB

All stacks are included (including Prometheus/Loki), you just have to install a custom-generated Agent package (I run Ubuntu Server 20, with the provided instructions it was as simple as a copy/paste).

After adding and setting up the basics, I now have 3 integrations: “Linux Server” (of course :wink: ), “Docker”, and “Grafana Agent” (not mandatory).
I was like a total virgin with all this stuff, but the steps are very straightforward with the provided web UI (don’t rely too much on docs, they sometime aren’t up to date …).
Don’t forget to download your new custom-generated Agent build, for each new integration.

Next, the not-so-tricky part: integration of CrowdSec metrics!

First, from Prometheus | CrowdSec, we can get 2 useful information: Crowdsec is exposing a Prometheus endpoint (on 6060), and some good guys published prebuilt dashboards (GitHub - crowdsecurity/grafana-dashboards: Grafana dashboards for Crowdsec monitoring using Prometheus).
Of course, my goal was to push metrics into the Grafana Cloud integrated Prometheus, not to install another one on my server :wink: .

Then, it took me a looong time to understand how Grafana Cloud works, after some readings and “trail&fail” I finally get my browser on this: Ship your metrics to Grafana Cloud without an integration | Grafana Cloud documentation.
Gotcha! I just followed the instructions for the “Hosted Prometheus” integration, except for downloading/running the binary: I already got an Agent installed, so I’d rather integrate the configuration into the existing file.
In short (full file below, way to the bottom), I just replaced the TCP port with 6060, and changed the job name with a sexy “crowdsec” :wink:
Then, in my “grafanacloud-xxx-prom”, I could see “cs_” metrics populating :smile_cat:

After importing the dashboards (I copied/pasted the raw JSON from GitHub), job done! :beers:

Here we are: a full Grafana monitoring for CrowdSec & Linux server, for free, with just a little agent running on my server.

Hope this will help!

My grafana-agent.yaml 
integrations:
  agent:
    enabled: true
    relabel_configs:
    - action: replace
      source_labels:
      - agent_hostname
      target_label: instance
  cadvisor:
    docker_only: true
    enabled: true
    instance: localhost:9090
    relabel_configs:
    - action: replace
      replacement: integrations/docker
      target_label: job
  node_exporter:
    enabled: true
    relabel_configs:
    - replacement: hostname
      target_label: instance
  prometheus_remote_write:
  - basic_auth:
      password: ***
      username: ***
    url: https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push
logs:
  configs:
  - clients:
    - basic_auth:
        password: ***
        username: ***
      url: https://logs-prod3.grafana.net/loki/api/v1/push
    name: integrations
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
    - job_name: integrations/node_exporter_journal_scrape
      journal:
        labels:
          instance: hostname
          job: integrations/node_exporter
        max_age: 24h
      relabel_configs:
      - source_labels:
        - __journal__systemd_unit
        target_label: unit
      - source_labels:
        - __journal__boot_id
        target_label: boot_id
      - source_labels:
        - __journal__transport
        target_label: transport
      - source_labels:
        - __journal_priority_keyword
        target_label: level
    - docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
      job_name: integrations/docker
      relabel_configs:
      - action: replace
        replacement: integrations/docker
        source_labels:
        - __meta_docker_container_id
        target_label: job
      - action: replace
        replacement: localhost:9090
        source_labels:
        - __meta_docker_container_id
        target_label: instance
      - regex: /(.*)
        source_labels:
        - __meta_docker_container_name
        target_label: container
      - source_labels:
        - __meta_docker_container_log_stream
        target_label: stream
    - job_name: integrations/agent
      journal:
        labels:
          instance: <hostname>
          job: integrations/agent
        max_age: 24h
      pipeline_stages:
      - match:
          action: drop
          drop_counter_reason: only_keep_grafana_agent_logs
          selector: '{unit!="grafana-agent.service"}'
      - regex:
          expression: (level=(?P<log_level>[\s]*debug|warn|info|error))
      - labels:
          level: log_level
      relabel_configs:
      - source_labels:
        - __journal__systemd_unit
        target_label: unit
    - job_name: integrations/agent
      pipeline_stages:
      - json:
          expressions:
            message: message
      - regex:
          expression: (level=(?P<log_level>[\s]*debug|warn|info|error))
      - labels:
          level: log_level
      - output:
          source: message
      relabel_configs:
      - source_labels:
        - computer
        target_label: instance
      - replacement: grafana-agent.service
        target_label: unit
      windows_events:
        bookmark_path: ./bookmark-application.xml
        eventlog_name: Application
        labels:
          job: integrations/agent
        use_incoming_timestamp: true
        xpath_query: '*[System[Provider[@Name=''Grafana Agent'']]]'
    target_config:
      sync_period: 10s
metrics:
  configs:
  - name: integrations
    remote_write:
    - basic_auth:
        password: ***
        username: ***
      url: https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push
  - name: hosted-prometheus
    scrape_configs:
      - job_name: crowdsec
        static_configs:
        - targets: ['localhost:6060']
    remote_write:
      - url: https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom/push
        basic_auth:
          username: ***
          password: ***
  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal
1 Like