Multi-Server setup - _all_ machines shown with old heartbeat every 30 Minutes

crowdsec
1

I have a multi-server setup 18 machines. LAPI-server with mariadb.
Since the latest update to v1.6.3-rpm-pragmatic-amd64-4851945a (all machines are on the same version, also may have happened in the previous version, but I just noticed it now), I often see some of them in ‘cscli machines list’ with a heartbeat of serveral hours behind. This clears within some minutes, but keeps happening again and again.
Until now I could not find any pattern for this behaviour. It look like all machines are affected.

Just now:
2024-10-11T07:52:00Z :heavy_check_mark: v1.6.3-rpm-pragmatic-amd64-4851945a Red Hat Enterprise Linux/8.10 password :warning: 6h30m17s

and about a minute later:
2024-10-11T07:53:59Z :heavy_check_mark: v1.6.3-rpm-pragmatic-amd64-4851945a Red Hat Enterprise Linux/8.10 password 15s

Also the version-string changes often from ‘v1.6.3-rpm-pragmatic-amd64-4851945a’ to ‘v1.6.3-rpm-pragmatic-amd64-4851945a-linux’ and back.

2

Do you have some machines that are using the same username and password that is generated by cscli machines add or cscli lapi register command?

3

Now that you are asking… I have indeed identified one clone of VM where still a crowdsec-firewall-bouncer.service was active/running and is now deactivated. For the rest I am not aware that I have any other clones running, but I will check.
But the old heartbeats still show up on machines where I am sure I never had any clones running.

4

More data: I started checking cscli machines list every 30 seconds and the heartbeat of all the machines is shown as old every 30 minutes.
Next I will have a look at the incoming packets from one of the machines.

5

The heartbeat communication from the machines looks identical at all times.

6

Hi @J-a-x-u,

I have the same problem with the same version as you,
Have you managed to find out what happened?

7

Hey all we found a bug/unforeseen issue in previous version that is now fixed but will be introduced in 1.6.5

the bug/unforeseen was introduced 6 months ago via: LAPI: detailed metrics endpoint (#2858) · srkoster/crowdsec@64e4ecd · GitHub

1 Like