Question about how to read the metrics

Hi there,

I just wanted to test crowdsec so I installed it on a local linux server. I haven’t done much with it yet and am still to isntall a bouncer. So far just looking at it and I found the metrics all show basically “unparsed lines”

shouldn’t it at least have parsed the curent logs?

cscli metrics
INFO[0000] Buckets Metrics:
+--------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------+---------------+-----------+--------------+--------+---------+
+--------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+-------------------+------------+--------------+----------------+------------------------+
|      SOURCE       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log |       1322 | -            |           1322 | -                      |
| /var/log/kern.log |        843 | -            |            843 | -                      |
| /var/log/messages |        871 | -            |            871 | -                      |
| /var/log/syslog   |      23574 | -            |          23574 | -                      |
+-------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+-------------------------------+-------+--------+----------+
|            PARSERS            | HITS  | PARSED | UNPARSED |
+-------------------------------+-------+--------+----------+
| child-crowdsecurity/sshd-logs |    10 | -      |       10 |
| crowdsecurity/iptables-logs   |  2529 | -      |     2529 |
| crowdsecurity/sshd-logs       |     2 | -      |        2 |
| crowdsecurity/syslog-logs     | 26610 |  26610 | -        |
+-------------------------------+-------+--------+----------+

I’m just wondering why it says hits 123 then parsed 0 !? Any help about how to read the metrics is highly appreciated

Hello @ovizii !

Yes, you are reading correctly and your logs are not correctly parsed by crowdsec.
Let’s look at it together :

Out of the 1322 lines that were read from the auth.log, none were parsed and none were poured to buckets.

Here we can see that while the syslog parser seems to work correctly (100% of logs were parsed), the others didn’t succeed at parsing any logs.

I hope this made the metrics more clear, now let’s figure out how to solve it!

  • Can you show me the list of parsers & scenarios that you currently have installed ? cscli list
  • Is /var/log/crowdsec.log containing error messages?

I guess the next step will be to use crowdsec to diagnose what can’t be parsed in the logs !

Let me know,

Thanks for looking into it, I must admit the problem could be me and not crowdsec though :slight_smile:

Here is an update output of cscli metrics:

INFO[0000] Buckets Metrics:
+--------------------------------+---------------+-----------+--------------+--------+---------+
|             BUCKET             | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/ssh-bf           | -             |         1 |            2 |      8 |       1 |
| crowdsecurity/ssh-bf_user-enum | -             | -         |            1 |      1 |       1 |
+--------------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+-------------------+------------+--------------+----------------+------------------------+
|      SOURCE       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log |       4296 |            8 |           4288 |                      9 |
| /var/log/kern.log |        931 | -            |            931 | -                      |
| /var/log/messages |       1022 | -            |           1022 | -                      |
| /var/log/syslog   |      71117 | -            |          71117 | -                      |
+-------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+--------------------------------+-------+--------+----------+
|            PARSERS             | HITS  | PARSED | UNPARSED |
+--------------------------------+-------+--------+----------+
| child-crowdsecurity/sshd-logs  |    34 |      8 |       26 |
| crowdsecurity/cdn-whitelist    |     3 |      3 | -        |
| crowdsecurity/dateparse-enrich |     8 |      8 | -        |
| crowdsecurity/geoip-enrich     |     8 |      8 | -        |
| crowdsecurity/iptables-logs    |  2793 | -      |     2793 |
| crowdsecurity/sshd-logs        |    10 |      8 |        2 |
| crowdsecurity/syslog-logs      | 77366 |  77366 | -        |
| crowdsecurity/whitelists       |     8 |      8 | -        |
+--------------------------------+-------+--------+----------+

This is the result of cscli list

INFO[0000] Loaded 13 collecs, 17 parsers, 20 scenarios, 3 post-overflow parsers
INFO[0000] unmanaged items : 32 local, 0 tainted
INFO[0000] PARSERS:
---------------------------------------------------------------------------------------------------------------------
 NAME                            �� STATUS    VERSION  LOCAL PATH
---------------------------------------------------------------------------------------------------------------------
 crowdsecurity/dateparse-enrich  ✔️  enabled  0.1      /etc/crowdsec/config/parsers/s02-enrich/dateparse-enrich.yaml
 crowdsecurity/iptables-logs     ✔️  enabled  0.1      /etc/crowdsec/config/parsers/s01-parse/iptables-logs.yaml
 crowdsecurity/sshd-logs         ✔️  enabled  0.1      /etc/crowdsec/config/parsers/s01-parse/sshd-logs.yaml
 crowdsecurity/whitelists        ✔️  enabled  0.1      /etc/crowdsec/config/parsers/s02-enrich/whitelists.yaml
 crowdsecurity/postfix-logs      ✔️  enabled  0.1      /etc/crowdsec/config/parsers/s01-parse/postfix-logs.yaml
 crowdsecurity/apache2-logs      ✔️  enabled  0.2      /etc/crowdsec/config/parsers/s01-parse/apache2-logs.yaml
 crowdsecurity/geoip-enrich      ✔️  enabled  0.2      /etc/crowdsec/config/parsers/s02-enrich/geoip-enrich.yaml
 crowdsecurity/http-logs         ✔️  enabled  0.2      /etc/crowdsec/config/parsers/s02-enrich/http-logs.yaml
 crowdsecurity/syslog-logs       ✔️  enabled  0.1      /etc/crowdsec/config/parsers/s00-raw/syslog-logs.yaml
 crowdsecurity/mysql-logs        ✔️  enabled  0.1      /etc/crowdsec/config/parsers/s01-parse/mysql-logs.yaml
 crowdsecurity/nginx-logs        ✔️  enabled  0.1      /etc/crowdsec/config/parsers/s01-parse/nginx-logs.yaml
---------------------------------------------------------------------------------------------------------------------
INFO[0000] SCENARIOS:
----------------------------------------------------------------------------------------------------------------------------------
 NAME                                       �� STATUS    VERSION  LOCAL PATH
----------------------------------------------------------------------------------------------------------------------------------
 crowdsecurity/http-sensitive-files         ✔️  enabled  0.2      /etc/crowdsec/config/scenarios/http-sensitive-files.yaml
 crowdsecurity/iptables-scan-multi_ports    ✔️  enabled  0.1      /etc/crowdsec/config/scenarios/iptables-scan-multi_ports.yaml
 crowdsecurity/http-crawl-non_statics       ✔️  enabled  0.2      /etc/crowdsec/config/scenarios/http-crawl-non_statics.yaml
 crowdsecurity/http-sqli-probing            ✔️  enabled  0.2      /etc/crowdsec/config/scenarios/http-sqli-probing.yaml
 crowdsecurity/ssh-bf                       ✔️  enabled  0.1      /etc/crowdsec/config/scenarios/ssh-bf.yaml
 crowdsecurity/http-probing                 ✔️  enabled  0.1      /etc/crowdsec/config/scenarios/http-probing.yaml
 crowdsecurity/http-backdoors-attempts      ✔️  enabled  0.2      /etc/crowdsec/config/scenarios/http-backdoors-attempts.yaml
 crowdsecurity/postfix-spam                 ✔️  enabled  0.1      /etc/crowdsec/config/scenarios/postfix-spam.yaml
 crowdsecurity/http-bad-user-agent          ✔️  enabled  0.2      /etc/crowdsec/config/scenarios/http-bad-user-agent.yaml
 crowdsecurity/http-path-traversal-probing  ✔️  enabled  0.2      /etc/crowdsec/config/scenarios/http-path-traversal-probing.yaml
 crowdsecurity/http-xss-probing             ✔️  enabled  0.2      /etc/crowdsec/config/scenarios/http-xss-probing.yaml
 crowdsecurity/mysql-bf                     ✔️  enabled  0.1      /etc/crowdsec/config/scenarios/mysql-bf.yaml
----------------------------------------------------------------------------------------------------------------------------------
INFO[0000] COLLECTIONS:
--------------------------------------------------------------------------------------------------------------------
 NAME                               �� STATUS    VERSION  LOCAL PATH
--------------------------------------------------------------------------------------------------------------------
 crowdsecurity/mysql                ✔️  enabled  0.1      /etc/crowdsec/config/collections/mysql.yaml
 crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/config/collections/linux.yaml
 crowdsecurity/nginx                ✔️  enabled  0.1      /etc/crowdsec/config/collections/nginx.yaml
 crowdsecurity/postfix              ✔️  enabled  0.1      /etc/crowdsec/config/collections/postfix.yaml
 crowdsecurity/sshd                 ✔️  enabled  0.1      /etc/crowdsec/config/collections/sshd.yaml
 crowdsecurity/apache2              ✔️  enabled  0.1      /etc/crowdsec/config/collections/apache2.yaml
 crowdsecurity/base-http-scenarios  ✔️  enabled  0.1      /etc/crowdsec/config/collections/base-http-scenarios.yaml
 crowdsecurity/iptables             ✔️  enabled  0.1      /etc/crowdsec/config/collections/iptables.yaml
--------------------------------------------------------------------------------------------------------------------
INFO[0000] POSTOVERFLOWS:
------------------------------------------------------------------------------------------------------------------------
 NAME                         �� STATUS    VERSION  LOCAL PATH
------------------------------------------------------------------------------------------------------------------------
 crowdsecurity/cdn-whitelist  ✔️  enabled  0.2      /etc/crowdsec/config/postoverflows/s01-whitelist/cdn-whitelist.yaml
------------------------------------------------------------------------------------------------------------------------

I am currently look at /var/log/crowdsec.log for errors - no errros found but hereq are a couple of lines that look suspicious (I am referring to the one about “no filename or filenames”

time="02-11-2020 13:39:35" level=warning msg="Loaded 13 scenarios"
time="02-11-2020 13:39:35" level=warning msg="Restoring buckets state from /tmp/crowdsec-buckets-dump-843844857"
time="02-11-2020 13:39:35" level=info msg="Restored 0 buckets from dump"
time="02-11-2020 13:39:35" level=info msg="Loading output profiles"
time="02-11-2020 13:39:35" level=info msg="Loading API client"
time="02-11-2020 13:39:35" level=info msg="API connector init"
time="02-11-2020 13:39:35" level=info msg="api load configuration: configuration loaded successfully (base:https://tmsov6x2n9.execute-api.eu-west-1.amazo
naws.com/v1/)"
time="02-11-2020 13:39:35" level=info msg="api signin: signed in successfuly"
time="02-11-2020 13:39:35" level=info msg="Opening file '/var/log/auth.log' (pattern:/var/log/auth.log)"
time="02-11-2020 13:39:35" level=info msg="Opening file '/var/log/syslog' (pattern:/var/log/syslog)"
time="02-11-2020 13:39:35" level=info msg="Opening file '/var/log/kern.log' (pattern:/var/log/kern.log)"
time="02-11-2020 13:39:35" level=info msg="Opening file '/var/log/messages' (pattern:/var/log/messages)"
time="02-11-2020 13:39:35" level=info msg="No filename or filenames, skipping empty item {Type:file Mode:tail Filename: Filenames:[] tail:<nil> Labels:ma
p[] Profiling:false}"
time="02-11-2020 13:39:35" level=info msg="Starting processing routines"
time="02-11-2020 13:39:35" level=warning msg="Starting processing data"
time="02-11-2020 13:39:35" level=info msg="starting (tail) reader file 0/4 : /var/log/auth.log"
time="02-11-2020 13:39:35" level=info msg="starting (tail) reader file 1/4 : /var/log/syslog"
time="02-11-2020 13:39:35" level=info msg="starting (tail) reader file 2/4 : /var/log/kern.log"
time="02-11-2020 13:39:35" level=info msg="starting (tail) reader file 3/4 : /var/log/messages"
time="02-11-2020 13:39:35" level=info msg="Started 4 routines for polling/read"
time="02-11-2020 13:39:35" level=info msg="Reload is finished"

on another note, I tried adding the log files for postfix since I had already installed the parsers and scenarios for postfix but even after reloading crowdsec service the metrics don’t reflect it :frowning:

edited /etc/crowdsec/config/acquis.yaml and added my code below the existing entries:

#Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/auth.log
filenames:
  - /var/log/auth.log
labels:
  type: syslog
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log /var/log/messages
filenames:
  - /var/log/syslog
  - /var/log/kern.log
  - /var/log/messages
labels:
  type: syslog
---
#manually added by ovi
filenames:
  - /var/log/mail.log
  - /var/log/mail.info
  - /var/log/mail.warn
  - /var/log/mail.err
labels:
  type: syslog
---

systemctl reload crowdsec

INFO[0000] Buckets Metrics:
+--------------------------------+---------------+-----------+--------------+--------+---------+
|             BUCKET             | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/ssh-bf           | -             |         1 |            2 |      8 |       1 |
| crowdsecurity/ssh-bf_user-enum | -             | -         |            1 |      1 |       1 |
+--------------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+-------------------+------------+--------------+----------------+------------------------+
|      SOURCE       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log |       4306 |            8 |           4298 |                      9 |
| /var/log/kern.log |        931 | -            |            931 | -                      |
| /var/log/messages |       1022 | -            |           1022 | -                      |
| /var/log/syslog   |      71290 | -            |          71290 | -                      |
+-------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+--------------------------------+-------+--------+----------+
|            PARSERS             | HITS  | PARSED | UNPARSED |
+--------------------------------+-------+--------+----------+
| child-crowdsecurity/sshd-logs  |    34 |      8 |       26 |
| crowdsecurity/cdn-whitelist    |     3 |      3 | -        |
| crowdsecurity/dateparse-enrich |     8 |      8 | -        |
| crowdsecurity/geoip-enrich     |     8 |      8 | -        |
| crowdsecurity/iptables-logs    |  2793 | -      |     2793 |
| crowdsecurity/sshd-logs        |    10 |      8 |        2 |
| crowdsecurity/syslog-logs      | 77549 |  77549 | -        |
| crowdsecurity/whitelists       |     8 |      8 | -        |
+--------------------------------+-------+--------+----------+

Did you try to reload or restart crowdsec service after adding this configuration ?

Yes, I mentioned it between those two code blocks above.

Does this look alright?

  systemctl status crowdsec
● crowdsec.service - Crowdwatch agent
   Loaded: loaded (/etc/systemd/system/crowdsec.service; disabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-11-20 16:32:40 CET; 26s ago
  Process: 27431 ExecStart=/usr/local/bin/crowdsec -c /etc/crowdsec/config/default.yaml (code=exited, status=0/SUCCESS)
  Process: 27449 ExecStartPost=/bin/sleep 0.1 (code=exited, status=0/SUCCESS)
  Process: 28035 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
 Main PID: 27443 (crowdsec)
    Tasks: 25 (limit: 4915)
   Memory: 46.9M
   CGroup: /system.slice/crowdsec.service
           └─27443 /usr/local/bin/crowdsec -c /etc/crowdsec/config/default.yaml

Nov 20 16:32:40 nas systemd[1]: Starting Crowdwatch agent...
Nov 20 16:32:40 nas systemd[1]: Started Crowdwatch agent.

Sorry didn’t see. Yes looks good.

When you reload crowdsec, you should new in /var/log/crowdsec.log :

time="20-11-2020 00:00:00" level=info msg="Starting tail of /var/log/mail.*"

And metrics about your new filenames (/var/log/mail.*).

1 Like