Greetings! Migrating from fail2ban…
I have a test machine running fresh ubuntu 20.04 install with nginx. I’ve configured basic authentication for nginx and verified that it’s prompting for credentials.
after numerous intentional auth failures I have yet to receive 403 error:
yet decisions list remains empty. What am I missing?
Thank you for the help!
cscli explain helped me figure this out:
cscli explain --file /var/log/nginx/error.log --type nginx
the IP I was submitting bad credentials from was within the 10.0.0.0/8 subnet in whitelists.yaml, hence it wasn’t getting blocked. Cool beans
The whitelist doesn’t seem to be the reason for failure to ban. I have three hosts running crowdsec, one with nginx and the nginx collection, the other with apache2 and the apache2 collection. The LAPI is running on a 3rd host.
I’ve commented out the private cidr ranges in /etc/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml on all 3 hosts and restarted crowdsec.
-During bf attempt on the nginx host, the following message was logged in /var/log/crowdsec.log:
time=“10-01-2022 16:56:07” level=error msg=“unable to collect sources from bucket: while extracting scope from bucket crowdsecurity/http-generic-bf: scope is Ip but Meta[source_ip] doesn’t exist”
-nothing is logged on apache2 host during bf.
-no decisions were made by LAPI
What steps should I take from here?
Thank you!
ssh-bf is working on nginx host… was promptly banned after several failed ssh logins:
+--------+----------+------------------+----------------------+--------+---------+----+--------+--------------------+----------+
| ID | SOURCE | SCOPE:VALUE | REASON | ACTION | COUNTRY | AS | EVENTS | EXPIRATION | ALERT ID |
+--------+----------+------------------+----------------------+--------+---------+----+--------+--------------------+----------+
| 106987 | crowdsec | Ip:10.0.0.2 | crowdsecurity/ssh-bf | ban | | 0 | 6 | 3h13m21.399405072s | 39 |
+--------+----------+------------------+----------------------+--------+---------+----+--------+--------------------+----------+
So maybe the problem is with the nginx bouncer?. Here is some debug info:
time="10-01-2022 19:19:50" level=debug msg="eval(evt.Meta.service == 'http' && evt.Meta.sub_type == 'auth_fail') = TRUE" cfg=winter-frost file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time="10-01-2022 19:19:50" level=debug msg="eval variables:" cfg=winter-frost file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time="10-01-2022 19:19:50" level=debug msg=" evt.Meta.service = 'http'" cfg=winter-frost file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time="10-01-2022 19:19:50" level=debug msg=" evt.Meta.sub_type = 'auth_fail'" cfg=winter-frost file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time="10-01-2022 19:19:50" level=debug msg="bucket 'crowdsecurity/http-generic-bf' is poured" cfg=winter-frost file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time="10-01-2022 19:19:50" level=debug msg="Last event to be poured, bucket overflow." bucket_id=empty-brook capacity=5 cfg=winter-frost file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf partition=0fd6cae51e1f374dbc284e4af378d5ce507a7734
time="10-01-2022 19:19:50" level=error msg="unable to collect sources from bucket: while extracting scope from bucket crowdsecurity/http-generic-bf: scope is Ip but Meta[source_ip] doesn't exist"
time="10-01-2022 19:19:50" level=debug msg="Adding overflow to blackhole (2022-01-10 19:19:35.231247563 +0000 UTC m=+26.654799649)" bucket_id=empty-brook capacity=5 cfg=winter-frost file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf partition=0fd6cae51e1f374dbc284e4af378d5ce507a7734
metrics output for nginx host:
INFO[10-01-2022 07:35:43 PM] Buckets Metrics:
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-crawl-non_statics | - | - | 2 | 2 | 2 |
| crowdsecurity/http-generic-bf | - | 1 | 2 | 8 | 1 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[10-01-2022 07:35:43 PM] Acquisition Metrics:
+--------------------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/auth.log | 4 | - | 4 | - |
| file:/var/log/nginx/access.log | 10 | 3 | 7 | 2 |
| file:/var/log/nginx/error.log | 8 | 8 | - | 8 |
| file:/var/log/syslog | 12 | - | 12 | - |
+--------------------------------+------------+--------------+----------------+------------------------+
INFO[10-01-2022 07:35:43 PM] Parser Metrics:
+--------------------------------------+------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+--------------------------------------+------+--------+----------+
| child-child-crowdsecurity/nginx-logs | 8 | 8 | - |
| child-crowdsecurity/http-logs | 9 | 3 | 6 |
| child-crowdsecurity/nginx-logs | 33 | 11 | 22 |
| crowdsecurity/dateparse-enrich | 3 | 3 | - |
| crowdsecurity/geoip-enrich | 3 | 3 | - |
| crowdsecurity/http-logs | 3 | - | 3 |
| crowdsecurity/nginx-logs | 18 | 11 | 7 |
| crowdsecurity/non-syslog | 18 | 18 | - |
| crowdsecurity/syslog-logs | 16 | 16 | - |
| crowdsecurity/whitelists | 11 | 11 | - |
+--------------------------------------+------+--------+----------+
Hello @Milton !
Thanks for the detailed issue 
By any chance, did you modify the nginx parser or such ?
The error you’re seeing is because (or at least it seems) the event that triggered the overflow didn’t contained a evt.Meta.source_ip which is expected to contain the offending IP.
Either you have a modified nginx parser that messes this up, or some weird logs, or we have an issue 
Would you mind sharing some log samples as well ?
Heyo, Thanks for responding!
I don’t think I’ve edited the parser. Here’s my md5 of /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml:
a01d2f87e0fa0a0b31460d9aea728041
here is an excerpt from /var/log/nginx/error.log:
2022/01/10 18:41:15 [error] 6866#6866: *26 user "admin" was not found in "/etc/apache2/.htpasswd", client: 192.168.1.1, server: _, request: "GET / HTTP/1.1", host: "10.0.0.2"
2022/01/10 19:19:44 [error] 901#901: *1 user "user1": password mismatch, client: 192.168.1.1, server: _, request: "GET / HTTP/1.1", host: "10.0.0.2"
nginx version:
nginx/1.18.0 (Ubuntu)
I also haven’t changed the log format of nginx from the default.
Found some log data related to the two errors in prior post, within /var/log/nginx/parser-dump.yaml:
- evt:
ExpectMode: 1
Stage: s01-parse
Line:
Raw: '2022/01/10 18:41:15 [error] 6866#6866: *26 user "admin" was not found
in "/etc/apache2/.htpasswd", client: 192.168.1.1, server: _, request:
"GET / HTTP/1.1", host: "10.0.0.2"'
Src: /var/log/nginx/error.log
time: 2022-01-10T20:22:53.087860797Z
Labels:
type: nginx
process: true
Module: file
Parsed:
message: '2022/01/10 18:41:15 [error] 6866#6866: *26 user "admin" was not
found in "/etc/apache2/.htpasswd", client: 192.168.1.1, server: _, request:
"GET / HTTP/1.1", host: "10.0.0.2"'
program: nginx
Time: 2022-01-10T20:22:53.090832795Z
Meta:
datasource_path: /var/log/nginx/error.log
datasource_type: file
success: true
- evt:
ExpectMode: 1
Stage: s01-parse
Line:
Raw: '2022/01/10 19:19:44 [error] 901#901: *1 user "user1": password mismatch,
client: 192.168.1.1, server: _, request: "GET / HTTP/1.1", host: "10.0.0.2"'
Src: /var/log/nginx/error.log
time: 2022-01-10T20:22:53.109077898Z
Labels:
type: nginx
process: true
Module: file
Parsed:
message: '2022/01/10 19:19:44 [error] 901#901: *1 user "user1": password mismatch,
client: 192.168.1.1, server: _, request: "GET / HTTP/1.1", host: "10.0.0.2"'
program: nginx
Time: 2022-01-10T20:22:53.111988101Z
Meta:
datasource_path: /var/log/nginx/error.log
datasource_type: file
success: true
Hello,
Checking how your logs are processed :
# cscli explain --file /tmp/xx.log --type nginx --verbose
line: 2022/01/10 18:41:15 [error] 6866#6866: *26 user "admin" was not found in "/etc/apache2/.htpasswd", client: 192.168.1.1, server: _, request: "GET / HTTP/1.1", host: "10.0.0.2"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| ├ 🔴 crowdsecurity/mysql-logs
| └ 🟢 crowdsecurity/nginx-logs (+4 ~1)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.username : admin
| └ create evt.Meta.sub_type : auth_fail
| └ create evt.Meta.username : admin
| └ create evt.Meta.service : http
├ s02-enrich
| ├ 🔴 crowdsecurity/dateparse-enrich
| ├ 🔴 crowdsecurity/geoip-enrich
| ├ 🔴 crowdsecurity/http-logs
| └ 🟢 crowdsecurity/whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
└ 🟢 crowdsecurity/http-generic-bf
line: 2022/01/10 19:19:44 [error] 901#901: *1 user "user1": password mismatch, client: 192.168.1.1, server: _, request: "GET / HTTP/1.1", host: "10.0.0.2"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| ├ 🔴 crowdsecurity/mysql-logs
| └ 🟢 crowdsecurity/nginx-logs (+4 ~1)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.username : user1
| └ create evt.Meta.sub_type : auth_fail
| └ create evt.Meta.username : user1
| └ create evt.Meta.service : http
├ s02-enrich
| ├ 🔴 crowdsecurity/dateparse-enrich
| ├ 🔴 crowdsecurity/geoip-enrich
| ├ 🔴 crowdsecurity/http-logs
| └ 🟢 crowdsecurity/whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
└ 🟢 crowdsecurity/http-generic-bf
It seems that the IP is indeed not captured, what might explain your issue. I’m going to have a further look at this.
Edit: It seems that it’s the server: _ part that is confusing the parser.
Edit2: opened an issue nginx error logs : `server: _` confuses parser · Issue #342 · crowdsecurity/hub · GitHub to track this
And here you go, it has been merged.
Please run cscli hub update and cscli hub upgrade and let me know, it should works 
Awesome! It appears to work now (for nginx)!
Thanks so much for this software!