Crowdesc service crashes when fetching decisions list

crowdsec
1

Hi,

I have a ubuntu server (18.0.4.6 LTS) running bigbluebutton (with ufw firewall) and crowdesc service installed for a while.

I noticed that crowdsec did not start properly after a system reboots. This is/was because of port 8080 that is used by tomcat:

lsof -i :8080
java    1303 tomcat8   63u  IPv6  28082      0t0  TCP *:http-alt (LISTEN)

This tomcat process is rather old and not needed, so I killed him manually and crowdesc service startet without problems in the past and i could also see decisions/blocked hosts, etc… - everything working fine.

I have the impression that with the last cowdsec update, this does not work any more!

When killing the tomcat process on port 8080, I can start crowdsec service but when fetching the decisions list with cscli decisions list the system needs a lot of time to respond and then tells me:

Unable to list decisions : performing request: Get "http://127.0.0.1:8080/v1/alerts?has_active_decision=true&include_capi=false&limit=100": could not get jwt token: Post "http://127.0.0.1:8080/v1/watchers/login": EOF

After that, the corwdsec service died and is not running any more!

The firewall settings (ufw) did not change - port 8080 is reachable/open. I do not have any idea, why the system is not wrking any more.

Any idea?

Thanks!

2

Is crowdsec running service running? on 8080?

3

Well, I am not sure on which port crowdsec is running. The default port I guess. It was never changed. When trying cscli decisions list it tries http://127.0.0.1:8080 (see above), so I guess it is running on port 8080.
Crowdsec service is running, but dies/stops after trying cscli decisions list.

Checking with losf says so

# lsof -i :8080
COMMAND    PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
crowdsec 29711 root   17u  IPv4 1398012      0t0  TCP localhost:http-alt (LISTEN)
crowdsec 29711 root   75u  IPv4 1398030      0t0  TCP localhost:38024->localhost:http-alt (ESTABLISHED)
crowdsec 29711 root   82u  IPv4 1399110      0t0  TCP localhost:http-alt->localhost:38024 (ESTABLISHED)
4

OK, now I noticed that the crowdsec agent stops working after some minute - without any request from my site. Checked the log:

time="15-02-2023 22:41:20" level=info msg="Adding leaky bucket" cfg=cool-sound file=/etc/crowdsec/scenarios/http-sensitive-files.yaml name=crowdsecurity/http-sensitive-files
time="15-02-2023 22:41:20" level=warning msg="Loaded 42 scenarios"
time="15-02-2023 22:41:20" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="15-02-2023 22:41:20" level=info msg="Adding file /var/log/nginx/access.log to datasources" type=file
time="15-02-2023 22:41:20" level=info msg="Adding file /var/log/nginx/bigbluebutton.access.log to datasources" type=file
time="15-02-2023 22:41:20" level=info msg="Adding file /var/log/nginx/error.log to datasources" type=file
time="15-02-2023 22:41:20" level=info msg="Adding file /var/log/auth.log to datasources" type=file
time="15-02-2023 22:41:20" level=info msg="Adding file /var/log/syslog to datasources" type=file
time="15-02-2023 22:41:20" level=info msg="Adding file /var/log/kern.log to datasources" type=file
time="15-02-2023 22:41:20" level=info msg="Starting processing data"
time="15-02-2023 22:42:59" level=error msg="capi pull top: while saving alert from crowdsecurity/community-blocklist: error creating alert : database is locked: unable to insert bulk"
time="15-02-2023 22:42:59" level=info msg="Start pull from CrowdSec Central API (interval: 1h57m26s once, then 2h0m0s)"
time="15-02-2023 22:43:01" level=error msg="Failed to update scenarios list for '4523babef7d342d88dbd330532b74088Ce7FUgfILwNRPSf5': unable to update machine in database: database is locked\n"
time="15-02-2023 22:43:01" level=fatal msg="starting outputs error : authenticate watcher (4523babef7d342d88dbd330532b74088Ce7FUgfILwNRPSf5): Post \"http://127.0.0.1:8080/v1/watchers/login\": **API error: incorrect Username or Password**"

What makes me wonder is: **API error: incorrect Username or Password** - never was changed by my side. Where can I check this credentials or reset them?
5

OK, what I did now after the mess with the credentials (still do not now why they did not work any more) is, that I removed/purged cowdsec service frome the host and reinstalled from scratch. Works!