Crowdsec Postgresql and Kubernetes

fbonalair · October 3, 2021, 12:58pm

Hello everyone, new user here,

First of all, thanks for the creator of Crowdsec, I was always looking for a more container compatible fail2ban and your solution seems interesting. So much so, that I’m on the way to right a Traefik bouncer to fit my need. (You can found the work in progress here).

I’ve manage to make Crowdsec work with my Kubernetes cluster … with the barebone docker container and my custom bouncer. My problem is managing to keep data between restart of container and more specifically bouncer API key. For that I want to use my Postgresql database (another container postgres:12.4-alpine):
I’ve followed the doc to create user and database with the following init script:

   init-crowdsec.sh : |
    #!/bin/bash
    set -e
    psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
        CREATE USER $CROWDSEC_USER PASSWORD '$CROWDSEC_PASSWORD';
        CREATE DATABASE $CROWDSEC_USER;
        GRANT ALL PRIVILEGES ON DATABASE $CROWDSEC_SCHEMA TO $CROWDSEC_USER;
    EOSQL

I have create a custom config.yaml from a copy of the container’s one:

common:
  daemonize: false
  pid_dir: /var/run/
  log_media: stdout
  log_level: info
  log_dir: /var/log/
  working_dir: .
config_paths:
  config_dir: /etc/crowdsec/
  data_dir: /var/lib/crowdsec/data/
  simulation_path: /etc/crowdsec/simulation.yaml
  hub_dir: /etc/crowdsec/hub/
  index_path: /etc/crowdsec/hub/.index.json
  notification_dir: /etc/crowdsec/notifications/
  plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
  acquisition_path: /etc/crowdsec/acquis.yaml
  parser_routines: 1
plugin_config:
  user: nobody
  group: nobody
cscli:
  output: human
db_config:
  log_level: info
  type: postgres
  user: ${CROWDSEC_USER}
  password: ${POSTGRES_PASSWORD}
  db_name: ${CROWDSEC_SCHEMA}
  host: postgresql.XXX.XXX.XXX.XXX # sanitized
  port: 5432
  sslmode: disable
  flush:
    max_items: 5000
    max_age: 7d
api:
  client:
    insecure_skip_verify: false
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  server:
    log_level: info
    listen_uri: 0.0.0.0:8080
    profiles_path: /etc/crowdsec/profiles.yaml
    online_client: # Central API credentials (to push signals and receive bad IPs)
      credentials_path: /etc/crowdsec/online_api_credentials.yaml
#    tls:
#      cert_file: /etc/crowdsec/ssl/cert.pem
#      key_file: /etc/crowdsec/ssl/key.pem
prometheus:
  enabled: true
  level: full
  listen_addr: 0.0.0.0
  listen_port: 6060

Checking the database’s schema, I see a couple of empty tables created (bouncer, alerts, decision etc) so I guess database connection if fine. Also, too be sure it is not a variable problem, I’ve both check env values with echo $ENV and directly write values in the file.

After a restart, from the logs it seems like crowdsec can’t access it’s API:

fbonalair:~$ kubectl -n kube-system logs crowdsec-agent-765d965f9f-542kx --previous 
Check if the container has already been started (ignore if agent is disabled)
time="03-10-2021 02:03:58 PM" level=info msg="push and pull to Central API disabled"
 localhost  127.0.0.1   2021-10-03T14:00:42+02:00  ✔️       v1.2.0-docker-0ecfe7568790a15791011da27eb24e96e7d4a39f 
Check if lapi need to register automatically an agent
registering agent localhost to lapi
AGENT PASSWORD : test
time="03-10-2021 02:03:58 PM" level=info msg="push and pull to Central API disabled"
time="03-10-2021 02:03:58 PM" level=fatal msg="unable to create machine: user 'localhost': user already exist"
time="03-10-2021 02:03:58 PM" level=warning msg="can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field)"
time="03-10-2021 02:03:58 PM" level=info msg="push and pull to Central API disabled"
time="03-10-2021 02:03:58 PM" level=info msg="Successfully registered to Central API (CAPI)"
time="03-10-2021 02:03:58 PM" level=info msg="Central API credentials dumped to '/etc/crowdsec/online_api_credentials.yaml'"
time="03-10-2021 02:03:58 PM" level=warning msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="03-10-2021 02:03:59 PM" level=info msg="Wrote new 138822 bytes index to /etc/crowdsec/hub/.index.json"
time="03-10-2021 02:03:59 PM" level=info msg="update for collection crowdsecurity/sshd available (currently:0.1, latest:0.2)"
time="03-10-2021 02:03:59 PM" level=info msg="dependency of crowdsecurity/linux : outdated collections crowdsecurity/sshd"
time="03-10-2021 02:03:59 PM" level=warning msg="crowdsecurity/syslog-logs : overwrite"
time="03-10-2021 02:03:59 PM" level=warning msg="crowdsecurity/geoip-enrich : overwrite"
time="03-10-2021 02:03:59 PM" level=warning msg="crowdsecurity/dateparse-enrich : overwrite"
time="03-10-2021 02:03:59 PM" level=warning msg="crowdsecurity/sshd-logs : overwrite"
time="03-10-2021 02:03:59 PM" level=warning msg="crowdsecurity/ssh-bf : overwrite"
time="03-10-2021 02:03:59 PM" level=info msg="crowdsecurity/ssh-slow-bf : OK"
time="03-10-2021 02:03:59 PM" level=info msg="Enabled scenarios : crowdsecurity/ssh-slow-bf"
time="03-10-2021 02:03:59 PM" level=warning msg="crowdsecurity/sshd : overwrite"
time="03-10-2021 02:03:59 PM" level=warning msg="crowdsecurity/sshd : overwrite"
time="03-10-2021 02:03:59 PM" level=warning msg="crowdsecurity/linux : overwrite"
time="03-10-2021 02:03:59 PM" level=info msg="📦 crowdsecurity/linux : updated"
time="03-10-2021 02:03:59 PM" level=info msg="Upgraded 1 items"
time="03-10-2021 02:03:59 PM" level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="03-10-2021 02:03:59 PM" level=info msg="crowdsecurity/whitelists : up-to-date"
time="03-10-2021 02:03:59 PM" level=info msg="Item 'crowdsecurity/whitelists' is up-to-date"
time="03-10-2021 02:03:59 PM" level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="03-10-2021 14:03:59" level=info msg="Crowdsec v1.2.0-docker-0ecfe7568790a15791011da27eb24e96e7d4a39f"
time="03-10-2021 14:03:59" level=info msg="Loading prometheus collectors"
time="03-10-2021 14:03:59" level=info msg="Loading CAPI pusher"
time="03-10-2021 14:03:59" level=info msg="start crowdsec api push (interval: 30s)"
time="03-10-2021 14:03:59" level=info msg="start crowdsec api pull (interval: 2h)"
time="03-10-2021 14:03:59" level=info msg="last CAPI pull is newer than 1h30, skip."
time="03-10-2021 14:03:59" level=info msg="start crowdsec api send metrics (interval: 30m)"
time="03-10-2021 14:03:59" level=info msg="Loading grok library /etc/crowdsec/patterns"
time="03-10-2021 14:03:59" level=info msg="Error machine login for  : ent: machine not found "
time="03-10-2021 14:03:59" level=info msg="127.0.0.1 - [Sun, 03 Oct 2021 14:03:59 CEST] \"POST /v1/watchers/login HTTP/1.1 401 837.822µs \"crowdsec/v1.2.0-docker-0ecfe7568790a15791011da27eb24e96e7d4a39f\" \""
time="03-10-2021 14:04:01" level=info msg="Loading enrich plugins"
time="03-10-2021 14:04:01" level=info msg="Successfully registered enricher 'GeoIpCity'"
time="03-10-2021 14:04:01" level=info msg="Successfully registered enricher 'GeoIpASN'"
time="03-10-2021 14:04:01" level=info msg="Successfully registered enricher 'IpToRange'"
time="03-10-2021 14:04:01" level=info msg="Successfully registered enricher 'reverse_dns'"
time="03-10-2021 14:04:01" level=info msg="Successfully registered enricher 'ParseDate'"
time="03-10-2021 14:04:01" level=info msg="Loading parsers 5 stages"
time="03-10-2021 14:04:01" level=info msg="Loaded 2 parser nodes" file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
time="03-10-2021 14:04:01" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
time="03-10-2021 14:04:01" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
time="03-10-2021 14:04:01" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
time="03-10-2021 14:04:01" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml
time="03-10-2021 14:04:01" level=info msg="Loaded 6 nodes, 3 stages"
time="03-10-2021 14:04:01" level=info msg="Loading postoverflow Parsers"
time="03-10-2021 14:04:01" level=info msg="Loaded 0 nodes, 0 stages"
time="03-10-2021 14:04:01" level=info msg="Loading 2 scenario files"
time="03-10-2021 14:04:01" level=info msg="Adding leaky bucket" cfg=autumn-glitter file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf
time="03-10-2021 14:04:01" level=info msg="Adding leaky bucket" cfg=polished-snow file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="03-10-2021 14:04:01" level=info msg="Adding leaky bucket" cfg=bitter-sea file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time="03-10-2021 14:04:01" level=info msg="Adding leaky bucket" cfg=muddy-water file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-10-2021 14:04:01" level=warning msg="Loaded 4 scenarios"
time="03-10-2021 14:04:01" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="03-10-2021 14:04:01" level=warning msg="No matching files for pattern /var/log/nginx/*.log" type=file
time="03-10-2021 14:04:01" level=warning msg="No matching files for pattern ./tests/nginx/nginx.log" type=file
time="03-10-2021 14:04:01" level=warning msg="No matching files for pattern /var/log/auth.log" type=file
time="03-10-2021 14:04:01" level=warning msg="No matching files for pattern /var/log/syslog" type=file
time="03-10-2021 14:04:01" level=warning msg="No matching files for pattern /var/log/apache2/*.log" type=file
time="03-10-2021 14:04:01" level=warning msg="Starting processing data"
time="03-10-2021 14:04:01" level=info msg="Error machine login for  : ent: machine not found "
time="03-10-2021 14:04:01" level=info msg="127.0.0.1 - [Sun, 03 Oct 2021 14:04:01 CEST] \"POST /v1/watchers/login HTTP/1.1 401 718.993µs \"crowdsec/v1.2.0-docker-0ecfe7568790a15791011da27eb24e96e7d4a39f\" \""
time="03-10-2021 14:04:01" level=fatal msg="starting outputs error : authenticate watcher (): Post \"http://127.0.0.1:8080/v1/watchers/login\": API error: ent: machine not found"
time="03-10-2021 14:04:01" level=fatal msg="starting outputs error : authenticate watcher (): Post \"http://127.0.0.1:8080/v1/watchers/login\": API error: ent: machine not found"

To be sure, I’ve tried sqlite as database, by mounting a volume for the folder /var/lib/crowdsec/data/. The container take much more time to start, work on first time but seems like I’ve got the same problem on restart:

fbonalair:~$ kubectl -n kube-system logs crowdsec-agent-765d965f9f-sx4cz --previous 
unable to retrieve container logs for containerd://71a0e4a1dd5707237e8498e599833e404798b71523ac3715253c69a245436888fbonalair@popos-workstation:~$ kubectl -n kube-system logs crowdsec-agent-765d965f9f-sx4cz --previous 
Check if the container has already been started (ignore if agent is disabled)
time="03-10-2021 02:50:56 PM" level=info msg="push and pull to Central API disabled"
 localhost  127.0.0.1   2021-10-03T14:49:59+02:00  ✔️       v1.2.0-docker-0ecfe7568790a15791011da27eb24e96e7d4a39f 
Check if lapi need to register automatically an agent
time="03-10-2021 02:50:56 PM" level=warning msg="can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field)"
time="03-10-2021 02:50:56 PM" level=info msg="push and pull to Central API disabled"
time="03-10-2021 02:50:56 PM" level=info msg="Successfully registered to Central API (CAPI)"
time="03-10-2021 02:50:56 PM" level=info msg="Central API credentials dumped to '/etc/crowdsec/online_api_credentials.yaml'"
time="03-10-2021 02:50:56 PM" level=warning msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="03-10-2021 02:50:57 PM" level=info msg="Wrote new 138822 bytes index to /etc/crowdsec/hub/.index.json"
time="03-10-2021 02:50:57 PM" level=info msg="dependency of crowdsecurity/linux : outdated collections crowdsecurity/sshd"
time="03-10-2021 02:50:57 PM" level=info msg="update for collection crowdsecurity/sshd available (currently:0.1, latest:0.2)"
time="03-10-2021 02:50:57 PM" level=warning msg="crowdsecurity/syslog-logs : overwrite"
time="03-10-2021 02:50:57 PM" level=warning msg="crowdsecurity/geoip-enrich : overwrite"
time="03-10-2021 02:50:57 PM" level=warning msg="crowdsecurity/dateparse-enrich : overwrite"
time="03-10-2021 02:50:57 PM" level=warning msg="crowdsecurity/sshd-logs : overwrite"
time="03-10-2021 02:50:57 PM" level=warning msg="crowdsecurity/ssh-bf : overwrite"
time="03-10-2021 02:50:57 PM" level=info msg="crowdsecurity/ssh-slow-bf : OK"
time="03-10-2021 02:50:57 PM" level=info msg="Enabled scenarios : crowdsecurity/ssh-slow-bf"
time="03-10-2021 02:50:57 PM" level=warning msg="crowdsecurity/sshd : overwrite"
time="03-10-2021 02:50:57 PM" level=warning msg="crowdsecurity/sshd : overwrite"
time="03-10-2021 02:50:57 PM" level=warning msg="crowdsecurity/linux : overwrite"
time="03-10-2021 02:50:57 PM" level=info msg="📦 crowdsecurity/linux : updated"
time="03-10-2021 02:50:57 PM" level=info msg="Upgraded 1 items"
time="03-10-2021 02:50:57 PM" level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="03-10-2021 02:50:57 PM" level=info msg="crowdsecurity/whitelists : up-to-date"
time="03-10-2021 02:50:57 PM" level=info msg="Item 'crowdsecurity/whitelists' is up-to-date"
time="03-10-2021 02:50:57 PM" level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="03-10-2021 14:50:57" level=info msg="Crowdsec v1.2.0-docker-0ecfe7568790a15791011da27eb24e96e7d4a39f"
time="03-10-2021 14:50:57" level=info msg="Loading prometheus collectors"
time="03-10-2021 14:50:57" level=info msg="Loading CAPI pusher"
time="03-10-2021 14:50:57" level=info msg="start crowdsec api push (interval: 30s)"
time="03-10-2021 14:50:57" level=info msg="start crowdsec api pull (interval: 2h)"
time="03-10-2021 14:50:57" level=info msg="start crowdsec api send metrics (interval: 30m)"
time="03-10-2021 14:50:57" level=info msg="last CAPI pull is newer than 1h30, skip."
time="03-10-2021 14:50:57" level=info msg="Loading grok library /etc/crowdsec/patterns"
time="03-10-2021 14:50:59" level=info msg="Loading enrich plugins"
time="03-10-2021 14:50:59" level=info msg="Successfully registered enricher 'GeoIpCity'"
time="03-10-2021 14:50:59" level=info msg="Successfully registered enricher 'GeoIpASN'"
time="03-10-2021 14:50:59" level=info msg="Successfully registered enricher 'IpToRange'"
time="03-10-2021 14:50:59" level=info msg="Successfully registered enricher 'reverse_dns'"
time="03-10-2021 14:50:59" level=info msg="Successfully registered enricher 'ParseDate'"
time="03-10-2021 14:50:59" level=info msg="Loading parsers 5 stages"
time="03-10-2021 14:50:59" level=info msg="Loaded 2 parser nodes" file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
time="03-10-2021 14:50:59" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
time="03-10-2021 14:50:59" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
time="03-10-2021 14:50:59" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
time="03-10-2021 14:50:59" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml
time="03-10-2021 14:50:59" level=info msg="Loaded 6 nodes, 3 stages"
time="03-10-2021 14:50:59" level=info msg="Loading postoverflow Parsers"
time="03-10-2021 14:50:59" level=info msg="Loaded 0 nodes, 0 stages"
time="03-10-2021 14:50:59" level=info msg="Loading 2 scenario files"
time="03-10-2021 14:50:59" level=info msg="Adding leaky bucket" cfg=bold-night file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf
time="03-10-2021 14:50:59" level=info msg="Adding leaky bucket" cfg=dry-meadow file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="03-10-2021 14:50:59" level=info msg="Adding leaky bucket" cfg=polished-snowflake file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time="03-10-2021 14:50:59" level=info msg="Adding leaky bucket" cfg=dark-pine file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-10-2021 14:50:59" level=warning msg="Loaded 4 scenarios"
time="03-10-2021 14:50:59" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="03-10-2021 14:50:59" level=warning msg="No matching files for pattern /var/log/nginx/*.log" type=file
time="03-10-2021 14:50:59" level=warning msg="No matching files for pattern ./tests/nginx/nginx.log" type=file
time="03-10-2021 14:50:59" level=warning msg="No matching files for pattern /var/log/auth.log" type=file
time="03-10-2021 14:50:59" level=warning msg="No matching files for pattern /var/log/syslog" type=file
time="03-10-2021 14:50:59" level=warning msg="No matching files for pattern /var/log/apache2/*.log" type=file
time="03-10-2021 14:50:59" level=warning msg="Starting processing data"
time="03-10-2021 14:50:59" level=info msg="Error machine login for  : ent: machine not found "
time="03-10-2021 14:50:59" level=info msg="127.0.0.1 - [Sun, 03 Oct 2021 14:50:59 CEST] \"POST /v1/watchers/login HTTP/1.1 401 593.839µs \"crowdsec/v1.2.0-docker-0ecfe7568790a15791011da27eb24e96e7d4a39f\" \""
time="03-10-2021 14:50:59" level=fatal msg="starting outputs error : authenticate watcher (): Post \"http://127.0.0.1:8080/v1/watchers/login\": API error: ent: machine not found"
time="03-10-2021 14:50:59" level=fatal msg="starting outputs error : authenticate watcher (): Post \"http://127.0.0.1:8080/v1/watchers/login\": API error: ent: machine not found"

Anybody have an idea were the problem can be?

PS: I have also tried to mount /etc/crowdsec/ to a Kubernetes volume but K8s mount an empty folder and Crowdsec don’t like it, even it I put the config.yaml aside and change its path with env var CONFIG_FILE

he2ss · October 3, 2021, 7:21pm

Hi @fbonalair,

We’re glad to see new contributor, if you face any issue don’t hesitate to ask for help here or using gitter.

For your issue, following this line:

time="03-10-2021 14:50:59" level=info msg="Error machine login for  : ent: machine not found "

As you can see the login is empty, so maybe the crowdsec agent didn’t register well. We need more details please (the env variables you configured on the docker image).

Just for curiosity, how did you install crowdsec on k8s ?
Because we just release the helm-chart of crowdsec installation in a k8s cluster (we just finished writing new blog post that will be posted soon).

So if you don’t know about this crowdsec chart, I think it will help you a lot . The chart documentation is still in progress, but if you are familiar with helm, you will get out.
And of course you can ask for help too.

fbonalair · October 13, 2021, 5:45pm

Sorry for the delay, I went in vacation in between.

Just for curiosity, how did you install crowdsec on k8s ?
Because we just release the helm-chart of crowdsec installation in a k8s cluster (we just finished writing new blog post that will be posted soon).

I’ve seen a helm chart somewhere, but for some reason it didn’t work. Though seeing the simple deployment of Crowdsec I didn’t debug and went the manual route. I would love to be notify when the helm blog post will be available.

As you can see the login is empty, so maybe the crowdsec agent didn’t register well. We need more details please (the env variables you configured on the docker image).

Here after the K8s yaml files:
Crowdsec internal yaml files:

apiVersion: v1
kind: ConfigMap
metadata:
  name: crowdsec-agent-files-conf
data:
  acquis.yaml: |
    filenames:
     - /var/log/auth.log
     - /var/log/syslog
    labels:
      type: syslog
  config.yaml: |
    common:
      daemonize: false
      pid_dir: /var/run/
      log_media: stdout
      log_level: info
      log_dir: /var/log/
      working_dir: .
    config_paths:
      config_dir: /etc/crowdsec/
      data_dir: /var/lib/crowdsec/data/
      simulation_path: /etc/crowdsec/simulation.yaml
      hub_dir: /etc/crowdsec/hub/
      index_path: /etc/crowdsec/hub/.index.json
      notification_dir: /etc/crowdsec/notifications/
      plugin_dir: /usr/local/lib/crowdsec/plugins/
    crowdsec_service:
      acquisition_path: /etc/crowdsec/acquis.yaml
      parser_routines: 1
    plugin_config:
      user: nobody
      group: nobody
    cscli:
      output: human
    db_config:
      log_level: info
      type: postgres
      user: ${CROWDSEC_USER}
      password: ${POSTGRES_PASSWORD}
      db_name: ${CROWDSEC_SCHEMA}
      host: postgresql.XXX.XXX.XXX.XXX # sanitized
      port: 5432
      sslmode: disable
      flush:
        max_items: 5000
        max_age: 7d
    api:
      client:
        insecure_skip_verify: false
        credentials_path: /etc/crowdsec/local_api_credentials.yaml
      server:
        log_level: info
        listen_uri: 0.0.0.0:8080
        profiles_path: /etc/crowdsec/profiles.yaml
        online_client: # Central API credentials (to push signals and receive bad IPs)
          credentials_path: /etc/crowdsec/online_api_credentials.yaml
    #    tls:
    #      cert_file: /etc/crowdsec/ssl/cert.pem
    #      key_file: /etc/crowdsec/ssl/key.pem
    prometheus:
      enabled: true
      level: full
      listen_addr: 0.0.0.0
      listen_port: 6060

The env values config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: crowdsec-agent-env-values
data:
  TZ: Europe/Paris
  CROWDSEC_USER: "crowdsec" 
  CROWDSEC_SCHEMA: "crowdsec"
  CONFIG_FILE: "/configMap/config.yaml"

And the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crowdsec-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crowdsec
      role: user-ban
      service: agent
  template:
    metadata:
      labels:
        app: crowdsec
        role: user-ban
        service: agent
    spec:
      restartPolicy: Always
      nodeSelector:
        kubernetes.io/arch: amd64 # FIXME changer le scheduler pour éviter de devoir faire ça manuellement: https://gist.github.com/squidpickles/dda268d9a444c600418da5e1641239af
      containers:
        - name: crowdsec-agent
          image: crowdsecurity/crowdsec:v1.2.0
          ports:
            - containerPort: 8080
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-passwords
                  key: CROWDSEC_PASSWORD
          envFrom:
            - configMapRef:
                name: crowdsec-agent-env-values
          livenessProbe:
            exec:
              command:
                - cscli
                - version
          readinessProbe:
            exec:
              command:
                - cscli
                - lapi
                - status
          volumeMounts: 
              # config files
            - name: config-files
              mountPath: /configMap/
              readOnly: true
      volumes:
        - name: config-files
          configMap:
            name: crowdsec-agent-files-conf

The secret for postgresql password is generated via:

kubectl create secret generic postgres-passwords \
  --from-literal=CROWDSEC_PASSWORD="$(openssl rand -base64 32)"

I hope it helps

he2ss · October 14, 2021, 3:04pm

Hi @fbonalair,

I hope you enjoyed your vacation.

The helm-chart is not well documented my bad, we’ll improve this one asap. I can confirm you the helm chart is working and you will be notified when the article will be available.

Thanks for sharing your configs, so I think I understood what’s happened.

The issue is that the Crowdsec docker image is built with default config which already includes /etc/crowdsec/config.yaml and a default SQLite database /var/lib/crowdsec/data/crowdsec.db. The docker_start.sh script tries to register a new machine but using the default config which interacting with the default database and not the Postgres one.
So when you try to run crowdsec with your new database, the docker_start runs and registers the local machine for the crowdsec agent, and when crowdsec run it used the Postgres one which is empty.

I just merged this fix. So waiting for the new official docker image that will be available, you can build your own docker image from the main branch and use it in your k8s cluster.

I hope it will solve your issue.

fbonalair · October 22, 2021, 7:01pm

Thanks for your work, I’ve manualy build the container and now the agent do seems to work. In postgres, table machines I do get a row for a local machine. (Though, password seems to be in clear, is it hashed or something?)

Though, I get another issue now, seems like bouncer registration isn’t working properly. My configuration haven’t changed from before. Here are my actions:

Registring a new bouncer with:

kubectl exec -it crowdsec-agent-57c997b5df-t2dxr -- cscli bouncers add test-bouncer
Api key for 'test-bouncer':

   99e8a4a5b2292687c87d5391c99d2faf

Please keep this key since you will not be able to retrieve it!

Connecting the crowdsec agent container to my host on port 8080:

kubectl port-forward -n kube-system crowdsec-agent-57c997b5df-t2dxr 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Requesting agent with API key:

curl  -H "X-Api-Key: 99e8a4a5b2292687c87d5391c99d2faf" -I localhost:8080/v1/decisions
HTTP/1.1 403 Forbidden
Content-Type: application/json; charset=utf-8
Date: Fri, 22 Oct 2021 18:55:56 GMT
Content-Length: 30

When I check in Postgres I have no row for bouncers:

SELECT t.*
          FROM public.bouncers t
          LIMIT 501
[2021-10-22 20:57:43] 0 rows retrieved in 28 ms (execution: 7 ms, fetching: 21 ms)

Though bouncer is actually registered:

kubectl -n kube-system exec -it crowdsec-agent-57c997b5df-t2dxr -- cscli bouncers list
------------------------------------------------------------------------------
 NAME             IP ADDRESS  VALID  LAST API PULL              TYPE  VERSION 
------------------------------------------------------------------------------
 test-bouncer                 ✔️      2021-10-22T20:54:55+02:00                
------------------------------------------------------------------------------

Am I missing something?

he2ss · October 25, 2021, 3:18pm

Hi @fbonalair

The password is hashed.

For the second issue, I think it’s because when you run cscli, you need to specify the crowdsec configuration, otherwise it will use the default one which specify a default path for the sqlite DB.

Try to use cscli with your config fil path :
kubectl exec -it crowdsec-agent-57c997b5df-t2dxr -- cscli -c /configMap/config.yaml bouncers add test-bouncer

fbonalair · October 25, 2021, 5:30pm

Lovely, it works now !
I’ve tagged your 2nd message as solution.

For the second issue, I think it’s because when you run cscli, you need to specify the crowdsec configuration, otherwise it will use the default one which specify a default path for the sqlite DB.

Shouldn’t the CLI also use the same env var as the application? Especially for people that use it often, that would be some flag to not to do every time.

he2ss · October 28, 2021, 12:48pm

Hi @fbonalair,

Nice.
For now the application don’t use an environment variable but launch crowdsec with the specified config file (here).
Actually it’s a good idea, I just opened an issue about that to make crowdsec and cscli supports env variables.

Thanks for your feedback.

Topic		Replies	Views
Crowdsec add Bouncer to Traefik in Docker container	7	1323	February 1, 2024
Running crowdsec on Rancher Kubernetes crowdsec	2	22	October 8, 2024
Docker running Traefik with CrowdSec Bouncer and PFsense as SE (LAPI)	3	416	May 21, 2024
Problems installing nginx-bouncer in k3s crowdsec	3	689	May 5, 2023
Massive Traefik Crowdsec Plugin/Middleware Kubernetes Issues crowdsec	6	487	April 15, 2024

Crowdsec Postgresql and Kubernetes

Related topics