Crowdsec not downloading blocklists?

tunloop · June 13, 2024, 3:19am

Just installed crowdsec and crowdsec-firewall-bouncer on Debian 12 and added it to the online console.

It shows as synced on the dashboard and I am subscribed to some blocklists. But none of the blocklists are showing up in the metrics?

I went through the installation instructions, and haven’t touched any config file in /etc/crowdsec.
Why isn’t this working??

Local API Decisions:
╭────────┬────────┬────────┬───────╮
│ Reason │ Origin │ Action │ Count │
├────────┼────────┼────────┼───────┤
╰────────┴────────┴────────┴───────╯

Local API Bouncers Metrics:
╭────────────────────────────────┬──────────────────────┬────────┬──────╮
│             Bouncer            │         Route        │ Method │ Hits │
├────────────────────────────────┼──────────────────────┼────────┼──────┤
│ cs-firewall-bouncer-x │ /v1/decisions/stream │ GET    │ 52   │
╰────────────────────────────────┴──────────────────────┴────────┴──────╯

Logs:

==> /var/log/crowdsec_api.log <==
time="2024-06-12T20:11:35-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:11:35 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 542.384µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:11:45-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:11:45 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 598.373µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:11:55-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:11:55 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 557.451µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:12:05-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:12:05 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 510.378µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:12:15-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:12:15 PDT] \"GET /v1/heartbeat HTTP/1.1 200 197.864µs \"crowdsec/v1.6.2-debian-pragmatic-amd64-x-linux\" \""
time="2024-06-12T20:12:15-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:12:15 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 432.13µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:12:25-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:12:25 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 529.959µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:12:35-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:12:35 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 527.607µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:12:45-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:12:45 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 587.182µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:12:55-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:12:55 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 548.765µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:13:05-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:13:05 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 536.545µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:13:15-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:13:15 PDT] \"GET /v1/heartbeat HTTP/1.1 200 246.534µs \"crowdsec/v1.6.2-debian-pragmatic-amd64-x-linux\" \""
time="2024-06-12T20:13:15-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:13:15 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 404.505µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:13:25-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:13:25 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 509.406µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:13:35-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:13:35 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 563.081µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:13:45-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:13:45 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 478.748µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:13:55-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:13:55 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 527.935µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:14:05-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:14:05 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 513.774µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""
time="2024-06-12T20:14:15-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:14:15 PDT] \"GET /v1/heartbeat HTTP/1.1 200 217.905µs \"crowdsec/v1.6.2-debian-pragmatic-amd64-x-linux\" \""
time="2024-06-12T20:14:15-07:00" level=info msg="127.0.0.1 - [Wed, 12 Jun 2024 20:14:15 PDT] \"GET /v1/decisions/stream HTTP/1.1 200 475.878µs \"crowdsec-firewall-bouncer/v0.0.28-debian-pragmatic-x\" \""

==> /var/log/crowdsec-firewall-bouncer.log <==
time="12-06-2024 20:04:08" level=info msg="Shutting down backend"
time="12-06-2024 20:04:08" level=info msg="removing 'crowdsec' table"
time="12-06-2024 20:04:08" level=info msg="removing 'crowdsec6' table"
time="12-06-2024 20:04:08" level=fatal msg="process terminated with error: received SIGTERM"
time="12-06-2024 20:04:15" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-debian-pragmatic-x"
time="12-06-2024 20:04:15" level=info msg="backend type : nftables"
time="12-06-2024 20:04:15" level=info msg="nftables initiated"
time="12-06-2024 20:04:15" level=info msg="Using API key auth"
time="12-06-2024 20:04:15" level=info msg="config is valid"
time="12-06-2024 20:04:15" level=info msg="Shutting down backend"
time="12-06-2024 20:04:15" level=info msg="removing 'crowdsec' table"
time="12-06-2024 20:04:15" level=info msg="removing 'crowdsec6' table"
time="12-06-2024 20:04:15" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-debian-pragmatic-x"
time="12-06-2024 20:04:15" level=info msg="backend type : nftables"
time="12-06-2024 20:04:15" level=info msg="nftables initiated"
time="12-06-2024 20:04:15" level=info msg="Using API key auth"
time="12-06-2024 20:04:15" level=info msg="Processing new and deleted decisions . . ."

==> /var/log/crowdsec.log <==
time="2024-06-12T20:04:14-07:00" level=warning msg="unable to initialize GeoIP: open /var/lib/crowdsec/data/GeoLite2-City.mmdb: no such file or directory"
time="2024-06-12T20:04:14-07:00" level=info msg="Loading grok library /etc/crowdsec/patterns"
time="2024-06-12T20:04:15-07:00" level=info msg="Loading enrich plugins"
time="2024-06-12T20:04:15-07:00" level=info msg="Successfully registered enricher 'GeoIpCity'"
time="2024-06-12T20:04:15-07:00" level=info msg="Successfully registered enricher 'GeoIpASN'"
time="2024-06-12T20:04:15-07:00" level=info msg="Successfully registered enricher 'IpToRange'"
time="2024-06-12T20:04:15-07:00" level=info msg="Successfully registered enricher 'reverse_dns'"
time="2024-06-12T20:04:15-07:00" level=info msg="Successfully registered enricher 'ParseDate'"
time="2024-06-12T20:04:15-07:00" level=info msg="Successfully registered enricher 'UnmarshalJSON'"
time="2024-06-12T20:04:15-07:00" level=info msg="Loading parsers from 0 files"
time="2024-06-12T20:04:15-07:00" level=info msg="Loaded 0 nodes from 0 stages"
time="2024-06-12T20:04:15-07:00" level=info msg="No postoverflow parsers to load"
time="2024-06-12T20:04:15-07:00" level=info msg="Loading 0 scenario files"
time="2024-06-12T20:04:15-07:00" level=info msg="Loaded 0 scenarios"
time="2024-06-12T20:04:15-07:00" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="2024-06-12T20:04:15-07:00" level=info msg="Adding file /var/log/nginx/error.log to datasources" type=file
time="2024-06-12T20:04:15-07:00" level=info msg="Adding file /var/log/nginx/access.log to datasources" type=file
time="2024-06-12T20:04:15-07:00" level=info msg="Starting processing data"
time="2024-06-12T20:04:15-07:00" level=info msg="Running journalctl command: /usr/bin/journalctl [journalctl --follow -n 0 _SYSTEMD_UNIT=ssh.service]" src="journalctl-_SYSTEMD_UNIT=ssh.service" type=journalctl
time="2024-06-12T20:04:15-07:00" level=info msg="capi metrics: sending"

iiAmLoz · June 13, 2024, 11:17am

And have you allowed up to two hours as the blocklists are synced the same as the community blocklist pull?

tunloop · June 14, 2024, 12:44am

Yep, and now its been sitting overnight. The dashboard shows its connecting very frequently.

But as of now, there are no IPs located in the nft ip set

# nft list table crowdsec
table ip crowdsec {
        set crowdsec-blacklists {
                type ipv4_addr
                flags timeout
        }

        chain crowdsec-chain-input {
                type filter hook input priority filter - 10; policy accept;
                ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
        }

        chain crowdsec-chain-forward {
                type filter hook forward priority filter - 10; policy accept;
                ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
        }
}

tunloop · June 14, 2024, 1:24am

Reinstalled the crowdsec packages and now the blocklists are populating in nft tables.
Not sure what the problem was, but its gone now.

Topic		Replies	Views
Did I have the rigth setup crowdsec	3	1066	August 17, 2022
V1.09 & cs-firewall-bouncer issue crowdsec	6	2639	October 15, 2021
[SOLVED] Ipset list crowdsec-blacklists and crowdsec6-blacklists are empty crowdsec	4	4231	April 27, 2022
crowdsec-nginx-bouncer not starting crowdsec	2	28	January 22, 2025
Bouncer not showing up crowdsec	1	275	April 2, 2024

Crowdsec not downloading blocklists?

Related topics