AppSec stops responding every few weeks essentially lockin me out

crowdsec
1

I am using traefik with crowdsec bouncer plugin which has the ability to send request to AppSec. By default this plugin is set to block traffic on AppSec unreachable and failure. And there lies the problem because about every 3 week AppSec will stop responding and essentially block all traffic. I can restart crowdsec docker container and everything is back to normal. This problem correlates with following log entries:

time="2025-05-31T18:56:13Z" level=error msg="Error performing request: Head \"http://0.0.0.0:8080/v1/decisions/stream\": dial tcp 0.0.0.0:8080: connect: cannot assign requested address" type=appsec
time="2025-05-31T18:56:13Z" level=error msg="Unauthorized request from '172.18.0.28:33508' (real IP = 10.0.0.30)" type=appsec

I had to restart to get my things back, so there is not much I can do to test right now until it happens again.

1 Like
2

same problem, restarting docker container fixed problem, eventually after 4-5 weeks problem returns

3

Hello,

Can you confirm the value of CrowdsecMode in the bouncer config ?
If set to appsec, then the bouncer will never contact the decision endpoint of LAPI, which will make Crowdsec think the bouncer is dead (for technical reasons, forwarded requests to the appsec are not taken into account when deciding if a bouncer is still alive or not).

If you have also bouncer auto-delete configured (db_config.flush.bouncers_autodelete in the crowdsec config), then the bouncer will be automatically deleted after some time (based on what you said, I’d guess you have a 3 weeks configuration).

In your case, the restart fixes it thanks to a side effect: you probably have set the bouncer API key in the env of the container, so it will be automatically recreated when the container starts.

For now, the “proper” workaround is to update the configuration of the bouncer:

  • Set CrowdsecMode to stream
  • Set CrowdsecAppsecEnabled to true

With this configuration, the bouncer will contact crowdsec to get the list of IPs to block, which will properly update its last query timestamp in the database, keeping it alive.

4

I already have the settings as you proposed:

    crowdsec:
      plugin:
        bouncer:
          enabled: true
          logLevel: INFO
          crowdsecMode: stream
          crowdsecAppsecEnabled: true
          crowdsecAppsecHost: crowdsec:7422
          crowdsecAppsecFailureBlock: true
          crowdsecAppsecUnreachableBlock: true
          crowdsecLapiKey: <API key>
          crowdsecLapiHost: crowdsec:8080
          crowdsecLapiScheme: http

When this happens, I am unable to get a response from crowdsec:8080 even when testing with curl. Normally I get 404.

5

Hello,

1.6.9 was released yesterday (the docker image will likely be released today), and it includes a mitigation for this: if the appsec cannot contact LAPI, but the key used by the bouncer was validated in the past, the appsec will still process the request.

This is not a proper fix, as we are still unsure what can cause this (apart from a slow database), but it should help in most cases.

6

Thank you. I do have port monitoring set up for 8080 now, so even if traefik does not out right ban me, I will still know when LAPI stops responding. I can report back and maybe leave it in that state for a day or two if we need to troubleshoot.

7

@blotus I started digging into the error specifics and one thing that never made sense was cannot assign requested address. Intuitively one would assume that this is the server/LAPI not being able to bind to 8080, but this message comes from the client side. So the only thing it can mean is ephemeral port exhaustion. My docker restarted at 4am this morning and there are already 800+ connections ESTABLISHED:

# netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.11:35698        0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:60048         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:46656         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:55006         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:53728         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:50920         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:49788         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:55730         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:59606         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:45748         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:50156         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:46298         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:45804         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:42612         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:53966         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:49310         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:49056         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:51168         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:52848         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:43820         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:44272         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:33206         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:42968         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:51704         127.0.0.1:8080          ESTABLISHED 1/crowdsec
tcp        0      0 127.0.0.1:45246         127.0.0.1:8080          ESTABLISHED 1/crowdsec
...

I am going to guess that something is not properly closing connections. I will let it run a few days and will report back.

8

We dont reuse the http client so go cannot manage the connections optimally so this will improve it.