I have a crowdsec container watching logs of a vaultwarden instance und on the docker host a bouncer wich as far as i can see seems i got to work fine (but i am not sure
I see the alerts, i see decisions, i see it also in the crowdsec dashboard but it seems the firewall rules are not set correctly.
558163 │ crowdsec │ Ip:91.5.xxx.xx │ Dominic-Wagner/vaultwarden-bf │ ban │ DE │ 3320 Deutsche Telekom AG │ 8 │ 3h47m43s │ 157
My iptables -L for DOCKER-USER:
Chain DOCKER-USER (1 references)
target prot opt source destination
CROWDSEC_CHAIN all – anywhere anywhere
RETURN all – anywhere anywhere
There is no DROP entry, thats all i can see, but I am not sure why.
Any help would be apreciated!
So the first thing to check if it exists in the ipset:
ipset list | grep <your_ip>
If it does exist are you using an upstream proxy like Cloudflare if so then unfortunately due to how proxied requests work the firewall remediation cannot act on layer 3/4 since all it can see cloudflare IP address (real IP is hidden in layer 7 header).
Yes the correct source ip is showing up here:
Name: crowdsec-blacklists-1
Type: hash:net
Revision: 7
Header: family inet hashsize 1024 maxelem 131072 timeout 300 bucketsize 12 initval 0xf5fc4a7d
Size in memory: 520
References: 1
Number of entries: 1
Members:
91.5.xx.25 timeout 2026
This is the correct ipadress from the source as shown in browser (https://www.whatsmyip.org/)
I forwarded it from cloudflare via NGINX Proxy
real_ip_header CF-Connecting-IP;
For info all this is running on Proxmox in an LXC Container, i dont know if the iptables work as usual in an LXC? But as far as i could google, they should.
You need any logfiles or something i can provide?
Maybe my understanding of this is wrong, but if I do real_ip_header CF-Connecting-IP then the destination should see the correct IP adress of the source? They show up in the vaultwarden logs and also in the ipset, so the firewall should also get it and be able to block?
If you are using Cloudflare with proxy enabled you can never use layer 3/4 remediation as at that layer all the firewall can see is Cloudflare IP address as in the OSI model layer 3/4 happens before layer 7 which is where nginx is setting the correct IP.
So would you have to use a remediation on layer 7 such as the nginx remediation.
Ok that makes sense then… sorry I guess I am not really deep enough in this networking…layer and all this. So there is absolutely no way to get crowdsec working behind cloudflare? Or is there another workaround how this could be done?
Is there a way to check what IP iptables really sees?
Yes like I said you would need to use a remediation on layer 7 such as Nginx | CrowdSec
It wont completely block the request but will still not return the content that the webserver would of return. Just to clarify this issue is not just a crowdsec issue (as if you wanted to do GEO blocking this would also prevent this for example) this is just how reverse proxies work but cloudflare markets it as “just put us infront” without actually explaining how this can effect the networking stack.
Is there a way to check what IP iptables really sees?
Yeah you can add a log rule to the INPUT chain but as said before you will just see cloudflare IP coming in.
Thanks for clarifying this! Just for learning purposes i tried to make an INPUT log rule, but syslog is always empty in the lxc - i tried to use ulogd to write the logfile to another location, but still nothing seems to be logged -so I gave up on this right now, and will remove the firewall bouncer and try to use the nginx/crowdsec bouncer.
Seems to me like also not really easy to configure, i’ll try my best Thanks for your help, really apreaciate it!
I had the same problem, crowdsec firewall bouncer can’t manage well iptables, nor nftables… In my case even nftables.
You never be sure it is working OK.
Other times crowdsec block legal traffic such as owncloud and others.
Every time you need to touch by hand a parser, scenario, white-list, etc, you need to touch complex yaml file, and if you fail, when whole crowdsec fail! Not only the scenario, parser, etc.
Thanks for your input. In this case, the issue stemmed from a misunderstanding of how reverse proxies like Cloudflare operate. While CrowdSec can block IPs at the network level, when traffic is routed through a reverse proxy, remediation at layer 3/4 is effectively bypassed — which can make it seem like the firewall bouncer isn’t working.
I understand the frustration, and CrowdSec can definitely have a learning curve. If you have any usable feedback that we may have missed in our documentation, please consider opening an issue here.
Also, a friendly reminder not to trust everything Mr. GPT says — we’ve seen users follow its advice blindly, but it can hallucinate, especially due to the lack of detailed public sources on CrowdSec.
I was not aware of the fact that iptables is blocking on a differnet layer, so with the approach of the NPM bouncer i got in running now for the first 3 services. The block at the NPM instead of the firewall is fine for me.Thanks again Loz!
With the next service (guacamole) I do still hava a little problem also related to this, as the logs show the origin IP, but also an internal one in the same line like this:
08:21:17.178 WARN o.a.g.r.auth.AuthenticationService - Authentication attempt from [87.158.xxx.177, 87.158.xxx.177, 172.xxx.0.2] for user “testuser” failed.
So it will always whitlist the whole line.
I tried made some changes in the parsers but i could not get it working. can i change the parser to only take the first IP, or to ignore the one starting with 172.x ?
pattern in the yaml is looking like that:
pattern: ‘%{TIME:timestamp} [%{DATA:thread}] %{LOGLEVEL:level} o.a.g.r.auth.AuthenticationService - Authentication attempt from [%{IPV4:source_ip}(?:, %{IPV4:source_ip})*] for user “%{GUAC_CUSTOMUSER:username}” failed’
Or another idea to deal with this as i was not able to stop logging the internal IPs for that service.
Thanls in advance!
