Dashboard - Queries and Clarifications

crowdsec
1

A CrodwSec instance has been installed on one of my server and have found no IP bans or no Dashboard Data … Seems very weird can you please help in this … Also find cscli metrics

±-------------------------------------±--------------±----------±-------------±-------±--------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
±-------------------------------------±--------------±----------±-------------±-------±--------+
| crowdsecurity/http-crawl-non_statics | 3 | - | 680 | 1359 | 677 |
| crowdsecurity/http-probing | - | - | 10 | 18 | 10 |
| crowdsecurity/ssh-bf | - | - | 7 | 8 | 7 |
| crowdsecurity/ssh-bf_user-enum | - | - | 7 | 7 | 7 |
±-------------------------------------±--------------±----------±-------------±-------±--------+
INFO[0000] Acquisition Metrics:
±--------------------------±-----------±-------------±---------------±-----------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
±--------------------------±-----------±-------------±---------------±-----------------------+
| /var/log/httpd/access_log | 2875 | 2875 | - | 1377 |
| /var/log/messages | 327 | - | 327 | - |
| /var/log/secure | 95 | 8 | 87 | 15 |
±--------------------------±-----------±-------------±---------------±-----------------------+
INFO[0000] Parser Metrics:
±-------------------------------±-----±-------±---------+
| PARSERS | HITS | PARSED | UNPARSED |
±-------------------------------±-----±-------±---------+
| child-crowdsecurity/http-logs | 8625 | 5970 | 2655 |
| child-crowdsecurity/sshd-logs | 311 | 8 | 303 |
| crowdsecurity/apache2-logs | 2875 | 2875 | - |
| crowdsecurity/dateparse-enrich | 2883 | 2883 | - |
| crowdsecurity/geoip-enrich | 2883 | 2883 | - |
| crowdsecurity/http-logs | 2875 | 2510 | 365 |
| crowdsecurity/non-syslog | 2875 | 2875 | - |
| crowdsecurity/sshd-logs | 67 | 8 | 59 |
| crowdsecurity/syslog-logs | 422 | 422 | - |
| crowdsecurity/whitelists | 2883 | 2883 | - |
±-------------------------------±-----±-------±---------+

2

Hello,

From what appears in your cscli metrics, I would say it’s normal (in the way that it’s not surprising nothing shows up in your dashboard).

  1. Regarding acquisition (logs being read/parsed)
±--------------------------±-----------±-------------±---------------±-----------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
±--------------------------±-----------±-------------±---------------±-----------------------+
| /var/log/httpd/access_log | 2875 | 2875 | - | 1377 |
| /var/log/messages | 327 | - | 327 | - |
| /var/log/secure | 95 | 8 | 87 | 15 |
±--------------------------±-----------±-------------±---------------±-----------------------+
  • 100% of your HTTP access logs are read and parsed, and nearly half of them are even being poured to buckets
  • none of your logs in /var/log/messages are being parsed (but it might be ok depending on what is there)
  • while most of your logs in /var/log/secure are not parsed, the ones that are, are being sent to buckets as well
  1. Regarding the buckets:
±-------------------------------------±--------------±----------±-------------±-------±--------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
±-------------------------------------±--------------±----------±-------------±-------±--------+
| crowdsecurity/http-crawl-non_statics | 3 | - | 680 | 1359 | 677 |
| crowdsecurity/http-probing | - | - | 10 | 18 | 10 |
| crowdsecurity/ssh-bf | - | - | 7 | 8 | 7 |
| crowdsecurity/ssh-bf_user-enum | - | - | 7 | 7 | 7 |
±-------------------------------------±--------------±----------±-------------±-------±--------+

Some buckets have been created, or are even existing instances right now, but none of them overflowed (you can see the column “OVERFLOWS” is always empty).
Events will be written in the database only once a bucket overflows.

So, from the metrics that are being shown here, I would say that it’s simply because no scenario have been triggered so far :slight_smile:

If you want to be sure it’s working correctly, you can simulate an attack yourself, such as a ssh-bruteforce or a web vulnerability scanner (based on the scenarios that seem enabled on your setup).

For the latest case, running nikto -host yourdomain.com might do the trick, and you should see overflows popping up, in both the cscli metrics output, and in the logs :wink:

3

As per your suggestion have used nikto tool and checked on the Server and could get the results on the dashboard … Now everything is working fine as expected … Thanks Thibault for you clear explanation …
INFO[0000] Buckets Metrics:
±------------------------------------------±--------------±----------±-------------±-------±--------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
±------------------------------------------±--------------±----------±-------------±-------±--------+
| crowdsecurity/http-crawl-non_statics | 1 | 28 | 942 | 3435 | 913 |
| crowdsecurity/http-path-traversal-probing | 1 | 21 | 22 | 90 | - |
| crowdsecurity/http-probing | 1 | 1 | 17 | 53 | 15 |
| crowdsecurity/http-sensitive-files | 1 | 8 | 9 | 60 | - |
| crowdsecurity/http-xss-probbing | - | 24 | 28 | 164 | 4 |
| crowdsecurity/ssh-bf | - | - | 8 | 9 | 8 |
| crowdsecurity/ssh-bf_user-enum | - | - | 8 | 8 | 8 |
±------------------------------------------±--------------±----------±-------------±-------±--------+
INFO[0000] Acquisition Metrics:
±--------------------------±-----------±-------------±---------------±-----------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
±--------------------------±-----------±-------------±---------------±-----------------------+
| /var/log/httpd/access_log | 5816 | 5816 | - | 3802 |
| /var/log/messages | 423 | - | 423 | - |
| /var/log/secure | 111 | 9 | 102 | 17 |
±--------------------------±-----------±-------------±---------------±-----------------------+
INFO[0000] Parser Metrics:
±---------------------------------±------±-------±---------+
| PARSERS | HITS | PARSED | UNPARSED |
±---------------------------------±------±-------±---------+
| child-crowdsecurity/http-logs | 17448 | 12334 | 5114 |
| child-crowdsecurity/sshd-logs | 363 | 9 | 354 |
| crowdsecurity/apache2-logs | 5816 | 5816 | - |
| crowdsecurity/cdn-whitelist | 7 | 7 | - |
| crowdsecurity/dateparse-enrich | 5825 | 5825 | - |
| crowdsecurity/geoip-enrich | 5825 | 5825 | - |
| crowdsecurity/http-logs | 5816 | 5248 | 568 |
| crowdsecurity/non-syslog | 5816 | 5816 | - |
| crowdsecurity/rdns | 7 | 7 | - |
| crowdsecurity/seo-bots-whitelist | 7 | 7 | - |
| crowdsecurity/sshd-logs | 78 | 9 | 69 |
| crowdsecurity/syslog-logs | 534 | 534 | - |
| crowdsecurity/whitelists | 5825 | 5825 | - |
±---------------------------------±------±-------±---------+
If you check for the overflows and the count is 28

Thanks & Regards
Neil