Firewall-bouncer work, but ipset is empty

1

Hi everybody
I have set up multi node installation as shown here https://www.crowdsec.net/blog/multi-server-setup
So i have one agent node and one lapi node
I have added and started iptables-bouncer on lapi side using

cscli bouncer add bouncer

When i get decision i see it on agent node,

INFO[20-11-2023 11:13:01 PM] Local Api Decisions:
+----------------------------------------+----------+--------+-------+
|                 REASON                 |  ORIGIN  | ACTION | COUNT |
+----------------------------------------+----------+--------+-------+
| crowdsecurity/nginx-req-limit-exceeded | crowdsec | ban    | 1     |
+----------------------------------------+----------+--------+-------+

but bouncer didn’t block banned ip
The iptables rule based on ipset created

[root@lb1 user]# iptables -nvL
Chain INPUT (policy ACCEPT 116K packets, 392M bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set crowdsec-blacklists src

I saw ipset - it’s created, but empty

[root@lb1 user]# ipset list
Name: crowdsec-blacklists
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536 timeout 300
Size in memory: 376
References: 1
Number of entries: 0
Members:

Name: crowdsec6-blacklists
Type: hash:net
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536 timeout 300
Size in memory: 1160
References: 1
Number of entries: 0
Members:

crowdsec version 1.4.1

2

here bouncer’s logs

time="20-11-2023 23:11:19" level=info msg="iptables clean-up : /usr/sbin/iptables -D INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="20-11-2023 23:11:19" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec-blacklists"
time="20-11-2023 23:11:19" level=info msg="Shutting down firewall-bouncer service"
time="20-11-2023 23:11:19" level=info msg="backend type : iptables"
time="20-11-2023 23:11:19" level=info msg="iptables for ipv4 initiated"
time="20-11-2023 23:11:19" level=info msg="iptables clean-up : /usr/sbin/iptables -D INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="20-11-2023 23:11:19" level=info msg="ipset 'crowdsec-blacklists' doesn't exist, skip"
time="20-11-2023 23:11:19" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec-blacklists"
time="20-11-2023 23:11:19" level=info msg="ipset 'crowdsec-blacklists' doesn't exist, skip"
time="20-11-2023 23:11:19" level=info msg="Checking existing set"
time="20-11-2023 23:11:19" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec-blacklists nethash timeout 300"
time="20-11-2023 23:11:20" level=info msg="Rule doesn't exist (/usr/sbin/iptables -C INPUT -m set --match-set crowdsec-blacklists src -j DROP)"
time="20-11-2023 23:11:20" level=info msg="iptables set-up : /usr/sbin/iptables -I INPUT -m set --match-set crowdsec-blacklists src -j DROP"
time="20-11-2023 23:11:20" level=info msg="iptables for ipv6 initiated"
time="20-11-2023 23:11:20" level=info msg="iptables clean-up : /usr/sbin/ip6tables -D INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="20-11-2023 23:11:20" level=info msg="ipset 'crowdsec6-blacklists' doesn't exist, skip"
time="20-11-2023 23:11:20" level=info msg="ipset clean-up : /usr/sbin/ipset -exist destroy crowdsec6-blacklists"
time="20-11-2023 23:11:20" level=info msg="ipset 'crowdsec6-blacklists' doesn't exist, skip"
time="20-11-2023 23:11:20" level=info msg="Checking existing set"
time="20-11-2023 23:11:20" level=info msg="ipset set-up : /usr/sbin/ipset -exist create crowdsec6-blacklists nethash timeout 300 family inet6"
time="20-11-2023 23:11:21" level=warning msg="iptables check command (/usr/sbin/ip6tables -C INPUT -m set --match-set crowdsec6-blacklists src -j DROP) failed : exit status 1"
time="20-11-2023 23:11:21" level=info msg="iptables set-up : /usr/sbin/ip6tables -I INPUT -m set --match-set crowdsec6-blacklists src -j DROP"
time="20-11-2023 23:11:21" level=info msg="Processing new and deleted decisions . . ."
3

Everything seems fine from these logs, are you applications within Docker? as the INPUT chain is bypassed for this and you need to configure DOCKER-USER within the bouncer config.

Re reading the top, did you configure your bouncer to communicate to the main lapi node?

4

Crowdsec runned with docker engine, but bounser worked on host
Yep, bouncer configured to communicate to the lapi


WARN[23-11-2023 12:13:34 PM] can't load CAPI credentials from '/etc/crowdsec/online_api_credentials.yaml' (missing field)
INFO[23-11-2023 12:13:34 PM] push and pull to Central API disabled
-------------------------------------------------------------------------------------------------------------------------------------------------------------
 NAME             IP ADDRESS    VALID  LAST API PULL              TYPE                       VERSION                                               AUTH TYPE
-------------------------------------------------------------------------------------------------------------------------------------------------------------
 stage1-firewall  *.*.*.*  ✔️      2023-11-23T15:13:32+03:00  crowdsec-firewall-bouncer  v0.0.24-rc1-0540b335e545b884d71e509ba9b437f9d67ee53c  api-key
-------------------------------------------------------------------------------------------------------------------------------------------------------------
5

Could you upgrade this to latest version?

6

Ready
I have installed crowdsec-firewall-bouncer-v0.0.29-rc1
Successfuly connected bouncer to lapi

stage1-firewall  *.*.*.*  ✔️      2023-11-24T14:14:17+03:00  crowdsec-firewall-bouncer  v0.0.29-rc1-39db9d6312259a693703627cc2c80ea9397a4906  api-key

Get Lapi decision

INFO[24-11-2023 11:50:33 AM] Local Api Decisions:
+----------------------------------------+----------+--------+-------+
|                 REASON                 |  ORIGIN  | ACTION | COUNT |
+----------------------------------------+----------+--------+-------+
| crowdsecurity/nginx-req-limit-exceeded | crowdsec | ban    | 1     |
+----------------------------------------+----------+--------+-------+

But ipset still empty

# ipset list
Name: crowdsec-blacklists
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 131072 timeout 300
Size in memory: 376
References: 1
Number of entries: 0
Members:
7

I find something
I can see decision only at agent machine

Both:

- cscli decisions list -a
- cscli metrics

on a Lapi host don’t see decisions

Is this ok?
Bouncer must be connected to lapi as i find in manual

ps
When i have connected bouncer to agent api - it’s work great

8

Looks like i found sollution
I haven’t disable api on agent nodes
Since i have run it in docker i set variable $DISABLE_LOCAL_API to true in docker_start.sh

9

I haven’t disable api on agent nodes

This shouldnt matter as like I asked previously did you configure the bouncer to communicate to the main node and thought you said yes?

10

Not sure but maybe this video from this channel will help. I used it to setup my multi server install.

Securing Multiple Webservers using Nginx proxy Manager with Crowdsec and Rsyslog.

Setup Crowdsec with Nginx Proxy Manager - Part 2 (Multi-server setup)

11

Yep, the bouncer was configured to work with main node, but untill i didn’t stop api at the agent node dicisions not applied