SSH attack causing crowdsec memory leak?

racitup · May 3, 2022, 3:21pm

Hi, I’ve just had to sort out a container which seemingly had a memory leak in crowdsec. I had to restart the container as the crowdsec service failed to restart with a timeout.
I’m not 100% certain it was caused by crowdsec, but didn’t seem to be anything else running on the container.

There’s an excerpt from the crowdsec log below which seems to point to the issue which has a corresponding auth.log.

The bits that seem worrying are the msg="stuck lines with failed_sent:299998 attempts:300000 type counts.

Any ideas on how I debug this further and prevent it from happening again?

On a related note, it seems crowdsec has a worryingly high virtual address space of 1.3GB (see screenshot). Can this be reduced?

Screen Shot 2022-05-03 at 16.10.16

2022/05/03 16:18:18 version: v1.2.1-debian-pragmatic-dd03d073558e380c283afe66942f537c3da647ff
2022/05/03 16:18:18 Codename: alphaga
2022/05/03 16:18:18 BuildDate: 2021-11-18_10:12:29
2022/05/03 16:18:18 GoVersion: 1.16.7
2022/05/03 16:18:18 Constraint_parser: >= 1.0, <= 2.0
2022/05/03 16:18:18 Constraint_scenario: >= 1.0, < 3.0
2022/05/03 16:18:18 Constraint_api: v1
2022/05/03 16:18:18 Constraint_acquis: >= 1.0, < 2.0

time="03-05-2022 07:46:44" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-slow-bf' (11 events over 19.388112314s) at 2022-05-03 07:46:43.943004671 +0100 BST m=+1837723.656452697"
time="03-05-2022 07:47:07" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-bf' (7 events over 19.339526453s) at 2022-05-03 07:47:05.655971249 +0100 BST m=+1837745.369419275"
time="03-05-2022 07:47:20" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-slow-bf_user-enum' (11 events over 48.619335067s) at 2022-05-03 07:47:19.221469605 +0100 BST m=+1837758.934917634"
time="03-05-2022 07:47:22" level=warning msg="stuck for 635.793134ms sending event to ab306ec2d8f14306a616e46b28ea5ba63e424779 (sigclosed:1 keymiss:1 failed_sent: 99997 attempts:100000)" cfg=proud-morning file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="03-05-2022 07:47:23" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-bf_user-enum' (9 events over 37.092166174s) at 2022-05-03 07:47:23.459918942 +0100 BST m=+1837763.173366971"
time="03-05-2022 07:47:52" level=warning msg="stuck for 774.868026ms sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 99998 attempts:100000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:47:52" level=warning msg="stuck for 1.136615189s sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 199998 attempts:200000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:47:52" level=warning msg="stuck for 1.2835585s sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent:299998 attempts:300000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:47:55" level=warning msg="stuck for 844.291623ms sending event to ab306ec2d8f14306a616e46b28ea5ba63e424779 (sigclosed:0 keymiss:1 failed_sent: 99998 attempts:100000)" cfg=proud-morning file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="03-05-2022 07:47:56" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-slow-bf' (11 events over 32.41204474s) at 2022-05-03 07:47:56.198555376 +0100 BST m=+1837795.912003391"
time="03-05-2022 07:48:12" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-bf' (7 events over 13.832507523s) at 2022-05-03 07:48:11.731275353 +0100 BST m=+1837811.444723373"
time="03-05-2022 07:48:26" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-bf_user-enum' (9 events over 32.449680198s) at 2022-05-03 07:48:25.356411646 +0100 BST m=+1837825.069859668"
time="03-05-2022 07:48:26" level=warning msg="stuck for 175.483036ms sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:1 keymiss:1 failed_sent: 99997 attempts:100000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:48:29" level=info msg="Ip 194.15.112.66 performed 'crowdsecurity/ssh-slow-bf_user-enum' (11 events over 33.106335165s) at 2022-05-03 07:48:28.421204653 +0100 BST m=+1837828.134652691"
time="03-05-2022 07:48:29" level=warning msg="stuck for 265.037444ms sending event to ab306ec2d8f14306a616e46b28ea5ba63e424779 (sigclosed:1 keymiss:1 failed_sent: 99997 attempts:100000)" cfg=proud-morning file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="03-05-2022 07:48:36" level=warning msg="stuck for 718.106497ms sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 99998 attempts:100000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 07:48:37" level=warning msg="stuck for 1.829922825s sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 199998 attempts:200000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="03-05-2022 08:02:58" level=warning msg="stuck for 657.215883ms sending event to ff28ee5fb2ff72db65d783775f08425420084ca5 (sigclosed:0 keymiss:1 failed_sent: 99998 attempts:100000)" cfg=crimson-paper file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum

alteredCoder · May 4, 2022, 10:01am

Hello, i can see that you have an old version of crowdsec.
Can you try to run the latest version and see if the problem still happens ?

racitup · May 5, 2022, 3:52am

I can upgrade but no I can’t reproduce the ssh attack. I doubt this part of the code has changed.

Why is it in a tight loop making 300,000 attempts at pouring a bucket?
It seems to me you should be sleeping for 100ms before another attempt.

github.com

crowdsecurity/crowdsec/blob/ad28a979e98436b468dd35101baa16335ad720a8/pkg/leakybucket/manager_run.go#L157

      
        
            	buckets.Bucket_map.Range(func(rkey, rvalue interface{}) bool {
            		key := rkey.(string)
            		val := rvalue.(*Leaky)
            		val.tomb.Kill(nil)
            		log.Infof("killed %s", key)
            		return true
            	})
            	return nil
            }
            
            
func PourItemToBucket(bucket *Leaky, holder BucketFactory, buckets *Buckets, parsed types.Event) (bool, error) {
            	var sent bool
            	var buckey = bucket.Mapkey
            	var err error
            
            
	sigclosed := 0
            	failed_sent := 0
            	attempts := 0
            	start := time.Now().UTC()
            
            
	for !sent {

racitup · May 11, 2022, 12:40pm

Any news? Issue 1519 reported here

racitup · May 16, 2022, 10:38am

requested context:
auth.log part1
auth log part2

Topic		Replies	Views
Troubleshoot high RAM consumption crowdsec	12	658	September 29, 2023
Is my setup working	4	1117	November 6, 2020
New Version of Crowdsec - Loses Connection to Console crowdsec	2	237	April 26, 2024
CS apparently not doing anything crowdsec	1	480	May 4, 2022
Why is virtual memory so large and how to reduce it? crowdsec	3	571	May 7, 2022

SSH attack causing crowdsec memory leak?

Related topics