Hi, I’ve just had to sort out a container which seemingly had a memory leak in crowdsec. I had to restart the container as the crowdsec service failed to restart with a timeout.
I’m not 100% certain it was caused by crowdsec, but didn’t seem to be anything else running on the container.
There’s an excerpt from the crowdsec log below which seems to point to the issue which has a corresponding auth.log.
The bits that seem worrying are the msg="stuck
lines with failed_sent:299998 attempts:300000
type counts.
Any ideas on how I debug this further and prevent it from happening again?
On a related note, it seems crowdsec has a worryingly high virtual address space of 1.3GB (see screenshot). Can this be reduced?
2022/05/03 16:18:18 version: v1.2.1-debian-pragmatic-dd03d073558e380c283afe66942f537c3da647ff
2022/05/03 16:18:18 Codename: alphaga
2022/05/03 16:18:18 BuildDate: 2021-11-18_10:12:29
2022/05/03 16:18:18 GoVersion: 1.16.7
2022/05/03 16:18:18 Constraint_parser: >= 1.0, <= 2.0
2022/05/03 16:18:18 Constraint_scenario: >= 1.0, < 3.0
2022/05/03 16:18:18 Constraint_api: v1
2022/05/03 16:18:18 Constraint_acquis: >= 1.0, < 2.0
time="03-05-2022 07:46:44" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-slow-bf' (11 events over 19.388112314s) at 2022-05-03 07:46:43.943004671 +0100 BST m=+1837723.656452697"
time="03-05-2022 07:47:07" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-bf' (7 events over 19.339526453s) at 2022-05-03 07:47:05.655971249 +0100 BST m=+1837745.369419275"
time="03-05-2022 07:47:20" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-slow-bf_user-enum' (11 events over 48.619335067s) at 2022-05-03 07:47:19.221469605 +0100 BST m=+1837758.934917634"
time="03-05-2022 07:47:22" level=warning msg="stuck for 635.793134ms sending event to ab306ec2d8f14306a616e46b28ea5ba63e424779 (sigclosed:1 keymiss:1 failed_sent: 99997 attempts:100000)" cfg=proud-morning file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="03-05-2022 07:47:23" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-bf_user-enum' (9 events over 37.092166174s) at 2022-05-03 07:47:23.459918942 +0100 BST m=+1837763.173366971"
time="03-05-2022 07:47:52" level=warning msg="stuck for 774.868026ms sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 99998 attempts:100000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:47:52" level=warning msg="stuck for 1.136615189s sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 199998 attempts:200000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:47:52" level=warning msg="stuck for 1.2835585s sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent:299998 attempts:300000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:47:55" level=warning msg="stuck for 844.291623ms sending event to ab306ec2d8f14306a616e46b28ea5ba63e424779 (sigclosed:0 keymiss:1 failed_sent: 99998 attempts:100000)" cfg=proud-morning file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="03-05-2022 07:47:56" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-slow-bf' (11 events over 32.41204474s) at 2022-05-03 07:47:56.198555376 +0100 BST m=+1837795.912003391"
time="03-05-2022 07:48:12" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-bf' (7 events over 13.832507523s) at 2022-05-03 07:48:11.731275353 +0100 BST m=+1837811.444723373"
time="03-05-2022 07:48:26" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-bf_user-enum' (9 events over 32.449680198s) at 2022-05-03 07:48:25.356411646 +0100 BST m=+1837825.069859668"
time="03-05-2022 07:48:26" level=warning msg="stuck for 175.483036ms sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:1 keymiss:1 failed_sent: 99997 attempts:100000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:48:29" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-slow-bf_user-enum' (11 events over 33.106335165s) at 2022-05-03 07:48:28.421204653 +0100 BST m=+1837828.134652691"
time="03-05-2022 07:48:29" level=warning msg="stuck for 265.037444ms sending event to ab306ec2d8f14306a616e46b28ea5ba63e424779 (sigclosed:1 keymiss:1 failed_sent: 99997 attempts:100000)" cfg=proud-morning file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="03-05-2022 07:48:36" level=warning msg="stuck for 718.106497ms sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 99998 attempts:100000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:48:37" level=warning msg="stuck for 1.829922825s sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 199998 attempts:200000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 08:02:58" level=warning msg="stuck for 657.215883ms sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 99998 attempts:100000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum