I’ve installed crowsec on a self-hosting server I have at home. I have a https://jellyfin.org/ server there and whenever I try to access, after a few requests, the rule crowdsecurity/http-crawl-non_statics is triggered. Two things I’d like to do :
add whitelist to ignore URLs that have /jellyfin/ in them
maybe improve crowdsecurity/http-crawl-non_statics detection to not consider this as “bad traffic” (I can provide some nginx logs for this), where should I start for this ?
Your whitelist seems correct to me (didn’t try it tho), did it work for you ?
You have a few solutions to do this :
Add a new whitelist file (probably the best, so you avoid tainting existing parsers) in /etc/crowdsec/config/parsers/s02-enrich/ :
name: my-custom-whitelist
description: my custom whitelists
whitelist:
reason: do not ban jellyfin users
expression:
- "'/jellyfin' in evt.Parsed.request"
Add a new section to your existing whitelist file (/etc/crowdsec/config/parsers/s02-enrich/whitelists.yaml) :
name: crowdsecurity/whitelists
description: "Whitelist events from private ipv4 addresses"
whitelist:
reason: "private ipv4 ranges"
...
---
name: my-custom-whitelist
description: my custom whitelists
whitelist:
reason: do not ban jellyfin users
expression:
- "'/jellyfin' in evt.Parsed.request"
Use the existing expression section of the existing whitelist :
(Please note that solutions 2 & 3 will “taint” your existing parser and it will prevent it from upgrading automatically in the future)
From what you said, crowdsecurity/http-crawl-non_statics seems to be quite prone to false positives, would you mind sharing some sample logs so we can improve it & reduce FP ?
You can usually find the scenarios in /etc/crowdsec/config/scenarios/ and yours more specifically in /etc/crowdsec/config/scenarios/http-crawl-non_statics.yaml. Please note this file will be a symlink to /etc/crowdsec/config/cscli/hub/scenarios/crowdsecurity/http-crawl-non_statics.yaml.
I’m not aware of any good tool for easy anonymization of logs, and quick search only lead me to dead projects, sorry
For some reason this (first option listed) turns up errors when I try it.
service crowdsec status
crowdsec.service - Crowdwatch agent
Loaded: loaded (/etc/systemd/system/crowdsec.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2020-11-07 15:43:05 GMT; 795ms ago
Process: 22442 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 30418 (code=exited, status=1/FAILURE)
Nov 07 15:43:02 xxx.com systemd[1]: Reloading Crowdwatch agent.
Nov 07 15:43:02 xxx.com systemd[1]: Reloaded Crowdwatch agent.
Nov 07 15:43:05 xxx.com systemd[1]: crowdsec.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 15:43:05 xxx.com systemd[1]: crowdsec.service: Unit entered failed state.
Nov 07 15:43:05 xxx.com systemd[1]: crowdsec.service: Failed with result ‘exit-code’.
Need to whitelist a couple static IPs which we use to access remote VPS.
Currently I’ve put the IPs in whitelists.yaml but, of course, it show up as tainted during updates.
As you can see here, ip needs to be at the same level as reason
Oh for heaven sake!
I should have been able to figure that out.
Donkey years ago I spent a couple days troubleshooting an issue which turned out to be with a text config file. Turned out the problem was the file couldn’t end in a carriage return. Reviewing the file, I never noticed the black/blank space below the last text line. That borked the whole program.