I was having some false positive bans for http-crawl-non_statics. Looking on the definition:
Is the “ressource” in line 4 a typo? I Changed it to “resource” and my false positives went away, but now I’m not sure if I’ve just broken the scenario 
Hi @n3dst4,
Thanks for your feedback. Indeed it’s a typo we left here.
So you need set it as ressource until we fix all the typos in the configuration.
About your false positive, could you be more specific please. If it trigger some good actors bots, you can install this collection with good actors whitelist : CrowdSec Hub
Hi, thanks for your reply. More detail on the false positives:
This is from crowdsec.log:
│time="05-04-2021 19:04:52" level=info msg="Ip 92.172.72.201 performed 'crowdsecurity/http-crawl-non_statics' (42 events over 893.58154
│9ms) at 2021-04-05 19:04:52.458772834 +0000 UTC m=+1472.396526257"
│time="05-04-2021 19:04:52" level=info msg="(045542e99c8f4e79a6e25c9af4e32383argP4i3JgRQtUCp1/crowdsec) crowdsecurity/http-crawl-non_st
│atics by ip 92.172.72.201 (FR) : 4h ban on Ip 92.172.72.201"
The IP in question was a friend of mine accessing the website and browsing some content (the times may be off in the logs compared to above, but this is representative)
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/dwarf%20king%27s%20surprise.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Carapace_Ground.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Ettercap_Supplicant_Portrait.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Falcon%20Portrait.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Ettercap_Warrior_Portrait.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Ettercap_Monk.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Drow_Ruffian.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/drakflagstones.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Drow%20Security%20Chief.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Falcon.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/dirtandrocks.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/dwark%20king%27s%20surprise%201st%20floor.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Falcon%20Full.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Campfire-2.webp HTTP/2.0" 304 0
│92.172.72.201 - - [05/Apr/2021:11:38:12 +0000] "GET /Simon/Ettercap_Temple_Guard.webp HTTP/2.0" 304 0
These are all static resources with 304 responses, and should not be triggering the http-crawl-non_statics scenario.
When I updated the scenario to say evt.Parsed.static_resource (not evt.Parsed.static_ressource) it stopped triggering the scenario every time someone uses the site.
Hello @n3dst4,
There is a typo in our configurations but ressource is actually good. This come from the http-logs parser as you can see here. So if you replace ressource by resource you must replace him also in the crowdsecurity/http-logs parser.
I think the false positive is another problem. I’m going to investigate with the log you pasted and i will come back to you.
EDIT: Can you please paste the output of cscli parsers list please? And if you have a custom parser for your http logs, can you paste it please?
It’s reporting a tainted http-crawl-non_statics because I still have that “resources” but I’m leaving it broken for now so I don’t block friendlies
ndc@ulthar ~ $ cscli parsers list
INFO[06-04-2021 10:58:10 AM] dependency issue crowdsecurity/base-http-scenarios : tainted scenarios crowdsecurity/http-crawl-non_statics, tainted.
INFO[06-04-2021 10:58:10 AM] dependency issue crowdsecurity/apache2 : sub collection crowdsecurity/base-http-scenarios is broken : tainted scenarios crowdsecurity/http-crawl-non_statics, tainted.
INFO[06-04-2021 10:58:10 AM] dependency issue crowdsecurity/apache2 : sub collection crowdsecurity/base-http-scenarios is broken : tainted scenarios crowdsecurity/http-crawl-non_statics, tainted.
INFO[06-04-2021 10:58:10 AM] dependency issue crowdsecurity/base-http-scenarios : tainted scenarios crowdsecurity/http-crawl-non_statics, tainted.
-------------------------------------------------------------------------------------------------------------
NAME 📦 STATUS VERSION LOCAL PATH
-------------------------------------------------------------------------------------------------------------
crowdsecurity/sshd-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
crowdsecurity/apache2-logs ✔️ enabled 0.5 /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml
crowdsecurity/geoip-enrich ✔️ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
crowdsecurity/whitelists ✔️ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/whitelists.yaml
crowdsecurity/syslog-logs ✔️ enabled 0.1 /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
crowdsecurity/dateparse-enrich ✔️ enabled 0.1 /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
crowdsecurity/http-logs ✔️ enabled 0.5 /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
crowdsecurity/iptables-logs ✔️ enabled 0.2 /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml
-------------------------------------------------------------------------------------------------------------
The relevant bit from acquis.yaml
---
filenames:
- /var/log/caddy/foundry.access_log
- /var/log/caddy/foundry2.access_log
- /var/log/caddy/cockpit.access_log
labels:
type: apache2
I’ve just repro’ed the false positive again, so I can give you the exact logs that trigger it:
I managed to reproduce the false positive with your log file, thanks
The problem was that .wav files were not considered as static resources.
I have opened a PR so this will be fixed soon.
Once the PR is merged, you will have to run sudo cscli hub update and sudo cscli parsers upgrade crowdsecurity/http-logs
1 Like
Excellent work! Thank you very much.
The pull request has been merged. Can you try please ?
Looks good. No false postives from some quick testing. Thank you again.