Metrics with Prometheus -

1

Hi

First of all thank you all for your effort to help out the users!!

I was able to set up CrowdSec and saw that I indeed have about on bounce a day (I have a homelab with about 40 container, providing media and nextcloud to my friends and family). Now I would like to have the metrics saved in prometheus, which does not work. Since I am new to prometheus too I have no idea if my CrowdSec config it wrong or the prometheus side.

The error message is: read tcp 172.29.0.15:53908->192.168.0.10:6060: read: connection reset by peer

The prometheus.yaml contains following snippet:

- job_name: crowdsec # Security related metrics
    http_sd_configs:
      - url: http://192.168.0.10:6060

I also tried

  - job_name: crowdsec # Security related metrics
    static_configs:
      - targets:
          - 192.168.0.10:6060
        labels:
          host: skippy

with the same result. The CrowdSec config.yml is fairly standard:

common:
  daemonize: false
  pid_dir: /var/run/
  log_media: stdout
  log_level: info
  log_dir: /var/log/
  working_dir: .
config_paths:
  config_dir: /etc/crowdsec/
  data_dir: /var/lib/crowdsec/data/
  simulation_path: /etc/crowdsec/simulation.yaml
  hub_dir: /etc/crowdsec/hub/
  index_path: /etc/crowdsec/hub/.index.json
  notification_dir: /etc/crowdsec/notifications/
  plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
  acquisition_path: /etc/crowdsec/acquis.yaml
  parser_routines: 1
plugin_config:
  user: nobody
  group: nobody
cscli:
  output: human
db_config:
  log_level: info
  type: sqlite
  db_path: /var/lib/crowdsec/data/crowdsec.db
  #user: 
  #password:
  #db_name:
  #host:
  #port:
  flush:
    max_items: 5000
    max_age: 7d
  use_wal: false
api:
  client:
    insecure_skip_verify: false
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  server:
    log_level: info
    listen_uri: 0.0.0.0:8080
    profiles_path: /etc/crowdsec/profiles.yaml
    trusted_ips: # IP ranges, or IPs which can have admin API access
      - 127.0.0.1
      - ::1
      - 192.168.0.0/24
    online_client: # Central API credentials (to push signals and receive bad IPs)
      credentials_path: /etc/crowdsec/online_api_credentials.yaml
      #credentials_path: /etc/crowdsec/online_api_credentials.yaml
#    tls:
#      cert_file: /etc/crowdsec/ssl/cert.pem
#      key_file: /etc/crowdsec/ssl/key.pem
prometheus:
  enabled: true
  level: full
  listen_addr: 192.168.0.10
  listen_port: 6060

Can somebody help me out here?

Disclaimer: I asked the same question in the prometheus forum. I will post the solution if I get it over there.

Thanks akrea

2

Hello @akrea !

To check prometheus on crowdsec’s side is running smoothly, can you try to curl 192.168.0.10:6060/metrics from your prometheus scrapper machine and see if you get a result ?

3

Also note have you allowed 6060 port through the host firewall?

Also it may help for us to know how is it setup

EG:
crowdsec installed on container host?
Prometheus is within a container so it most likely cant connect to 192.168 address. So you would change crowdsec prometheus url to be <docker_gateway_ip>

4

Hello thibault

I will answer iiAmLoz first:
I have a docker-stack of about 50 containers most of them in the same network. The ones open to the internet are behind a reverse proxy with TFA-app. Where necessary ports are open to the LAN so the containers can communicate with each other. Except for the the crowdsec-bouncer and some system relevant programs, everything is dockerized. I’m not very literate on firewall topics. So fasr I never had a firewall issue with any container.

For your question:
Now curl 192.168.0.10:6060/metrics executed in the crowdsec container gives me bash: curl: command not found as it is obvousliy not installed on the (I suspect) alpine version of linux in the image.

On a hunch I also tried cscli metrics and got this message:
FATA[30-12-2022 11:22:08 AM] failed to fetch prometheus metrics : executing GET request for URL "http://192.168.0.10:6060/metrics" failed: Get "http://192.168.0.10:6060/metrics": EOF
Note, that when I disable prometheus I get the cli metrics. FYI Prometheus is on port 9090.

I then went into the logs of crowdsec and saw this: prometheus: listen tcp 192.168.0.10:6060: bind: cannot assign requested address.
So there is probably something wrong with my crowdsec ports config in docker-compose, but I wouldn’t know how to change that (seems to be rather obvious but I don’t see it):

    crowdsec:
        <<: *common-keys-core
        image: crowdsecurity/crowdsec
        container_name: crowdsec
        ports:
            - "$CROWDSEC_API_PORT:8080"
            - "6060:6060" # For metrics export to Prometheus database.
        environment:
            ...
        volumes:
            ...

any ideas?

5

Hi @akrea ,
Crowdsec container is not aware of the host’s local IP 192.168.0.10 unless you set network_mode to host, so CS could not set it as listening IP for promethues.

On config.yml , you could try to expose it on :

  • 0.0.0.0 (as you did for API)
  • Or, on Local IP of your container, something like 172.X.X.X (Prometheus must be on the same network to reach it)

Or you can set network_mode to host for crowdsec container on your compose file.

6

Sorry I did not come back on this one. Many things on my to do list. For now I use the online-dashboard of crowdsec. Maybe at a later time. Will post again if I come around to use it.

Thank you anyway for your help!!

1 Like