I am using traefik with crowdsec bouncer plugin which has the ability to send request to AppSec. By default this plugin is set to block traffic on AppSec unreachable and failure. And there lies the problem because about every 3 week AppSec will stop responding and essentially block all traffic. I can restart crowdsec docker container and everything is back to normal. This problem correlates with following log entries:
time="2025-05-31T18:56:13Z" level=error msg="Error performing request: Head \"http://0.0.0.0:8080/v1/decisions/stream\": dial tcp 0.0.0.0:8080: connect: cannot assign requested address" type=appsec
time="2025-05-31T18:56:13Z" level=error msg="Unauthorized request from '172.18.0.28:33508' (real IP = 10.0.0.30)" type=appsec
I had to restart to get my things back, so there is not much I can do to test right now until it happens again.
Can you confirm the value of CrowdsecMode in the bouncer config ?
If set to appsec, then the bouncer will never contact the decision endpoint of LAPI, which will make Crowdsec think the bouncer is dead (for technical reasons, forwarded requests to the appsec are not taken into account when deciding if a bouncer is still alive or not).
If you have also bouncer auto-delete configured (db_config.flush.bouncers_autodelete in the crowdsec config), then the bouncer will be automatically deleted after some time (based on what you said, I’d guess you have a 3 weeks configuration).
In your case, the restart fixes it thanks to a side effect: you probably have set the bouncer API key in the env of the container, so it will be automatically recreated when the container starts.
For now, the “proper” workaround is to update the configuration of the bouncer:
Set CrowdsecMode to stream
Set CrowdsecAppsecEnabled to true
With this configuration, the bouncer will contact crowdsec to get the list of IPs to block, which will properly update its last query timestamp in the database, keeping it alive.
1.6.9 was released yesterday (the docker image will likely be released today), and it includes a mitigation for this: if the appsec cannot contact LAPI, but the key used by the bouncer was validated in the past, the appsec will still process the request.
This is not a proper fix, as we are still unsure what can cause this (apart from a slow database), but it should help in most cases.
Thank you. I do have port monitoring set up for 8080 now, so even if traefik does not out right ban me, I will still know when LAPI stops responding. I can report back and maybe leave it in that state for a day or two if we need to troubleshoot.
@blotus I started digging into the error specifics and one thing that never made sense was cannot assign requested address. Intuitively one would assume that this is the server/LAPI not being able to bind to 8080, but this message comes from the client side. So the only thing it can mean is ephemeral port exhaustion. My docker restarted at 4am this morning and there are already 800+ connections ESTABLISHED:
# netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.11:35698 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:60048 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:46656 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:55006 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:53728 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:50920 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:49788 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:55730 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:59606 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:45748 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:50156 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:46298 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:45804 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:42612 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:53966 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:49310 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:49056 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:51168 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:52848 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:43820 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:44272 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:33206 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:42968 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:51704 127.0.0.1:8080 ESTABLISHED 1/crowdsec
tcp 0 0 127.0.0.1:45246 127.0.0.1:8080 ESTABLISHED 1/crowdsec
...
I am going to guess that something is not properly closing connections. I will let it run a few days and will report back.