ovizii
November 1, 2020, 4:50pm
1
Hi there,
I just wanted to test crowdsec so I installed it on a local linux server. I haven’t done much with it yet and am still to isntall a bouncer. So far just looking at it and I found the metrics all show basically “unparsed lines”
shouldn’t it at least have parsed the curent logs?
cscli metrics
INFO[0000] Buckets Metrics:
+--------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------+---------------+-----------+--------------+--------+---------+
+--------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+-------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log | 1322 | - | 1322 | - |
| /var/log/kern.log | 843 | - | 843 | - |
| /var/log/messages | 871 | - | 871 | - |
| /var/log/syslog | 23574 | - | 23574 | - |
+-------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+-------------------------------+-------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+-------------------------------+-------+--------+----------+
| child-crowdsecurity/sshd-logs | 10 | - | 10 |
| crowdsecurity/iptables-logs | 2529 | - | 2529 |
| crowdsecurity/sshd-logs | 2 | - | 2 |
| crowdsecurity/syslog-logs | 26610 | 26610 | - |
+-------------------------------+-------+--------+----------+
I’m just wondering why it says hits 123 then parsed 0 !? Any help about how to read the metrics is highly appreciated
Hello @ovizii !
Yes, you are reading correctly and your logs are not correctly parsed by crowdsec.
Out of the 1322 lines that were read from the auth.log, none were parsed and none were poured to buckets.
Here we can see that while the syslog parser seems to work correctly (100% of logs were parsed), the others didn’t succeed at parsing any logs.
I hope this made the metrics more clear, now let’s figure out how to solve it!
Can you show me the list of parsers & scenarios that you currently have installed ? cscli list
Is /var/log/crowdsec.log containing error messages?
I guess the next step will be to use crowdsec to diagnose what can’t be parsed in the logs !
Let me know,
ovizii
November 6, 2020, 2:05pm
3
Thanks for looking into it, I must admit the problem could be me and not crowdsec though
Here is an update output of cscli metrics:
INFO[0000] Buckets Metrics:
+--------------------------------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/ssh-bf | - | 1 | 2 | 8 | 1 |
| crowdsecurity/ssh-bf_user-enum | - | - | 1 | 1 | 1 |
+--------------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+-------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log | 4296 | 8 | 4288 | 9 |
| /var/log/kern.log | 931 | - | 931 | - |
| /var/log/messages | 1022 | - | 1022 | - |
| /var/log/syslog | 71117 | - | 71117 | - |
+-------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+--------------------------------+-------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+--------------------------------+-------+--------+----------+
| child-crowdsecurity/sshd-logs | 34 | 8 | 26 |
| crowdsecurity/cdn-whitelist | 3 | 3 | - |
| crowdsecurity/dateparse-enrich | 8 | 8 | - |
| crowdsecurity/geoip-enrich | 8 | 8 | - |
| crowdsecurity/iptables-logs | 2793 | - | 2793 |
| crowdsecurity/sshd-logs | 10 | 8 | 2 |
| crowdsecurity/syslog-logs | 77366 | 77366 | - |
| crowdsecurity/whitelists | 8 | 8 | - |
+--------------------------------+-------+--------+----------+
This is the result of cscli list
INFO[0000] Loaded 13 collecs, 17 parsers, 20 scenarios, 3 post-overflow parsers
INFO[0000] unmanaged items : 32 local, 0 tainted
INFO[0000] PARSERS:
---------------------------------------------------------------------------------------------------------------------
NAME �� STATUS VERSION LOCAL PATH
---------------------------------------------------------------------------------------------------------------------
crowdsecurity/dateparse-enrich ✔️ enabled 0.1 /etc/crowdsec/config/parsers/s02-enrich/dateparse-enrich.yaml
crowdsecurity/iptables-logs ✔️ enabled 0.1 /etc/crowdsec/config/parsers/s01-parse/iptables-logs.yaml
crowdsecurity/sshd-logs ✔️ enabled 0.1 /etc/crowdsec/config/parsers/s01-parse/sshd-logs.yaml
crowdsecurity/whitelists ✔️ enabled 0.1 /etc/crowdsec/config/parsers/s02-enrich/whitelists.yaml
crowdsecurity/postfix-logs ✔️ enabled 0.1 /etc/crowdsec/config/parsers/s01-parse/postfix-logs.yaml
crowdsecurity/apache2-logs ✔️ enabled 0.2 /etc/crowdsec/config/parsers/s01-parse/apache2-logs.yaml
crowdsecurity/geoip-enrich ✔️ enabled 0.2 /etc/crowdsec/config/parsers/s02-enrich/geoip-enrich.yaml
crowdsecurity/http-logs ✔️ enabled 0.2 /etc/crowdsec/config/parsers/s02-enrich/http-logs.yaml
crowdsecurity/syslog-logs ✔️ enabled 0.1 /etc/crowdsec/config/parsers/s00-raw/syslog-logs.yaml
crowdsecurity/mysql-logs ✔️ enabled 0.1 /etc/crowdsec/config/parsers/s01-parse/mysql-logs.yaml
crowdsecurity/nginx-logs ✔️ enabled 0.1 /etc/crowdsec/config/parsers/s01-parse/nginx-logs.yaml
---------------------------------------------------------------------------------------------------------------------
INFO[0000] SCENARIOS:
----------------------------------------------------------------------------------------------------------------------------------
NAME �� STATUS VERSION LOCAL PATH
----------------------------------------------------------------------------------------------------------------------------------
crowdsecurity/http-sensitive-files ✔️ enabled 0.2 /etc/crowdsec/config/scenarios/http-sensitive-files.yaml
crowdsecurity/iptables-scan-multi_ports ✔️ enabled 0.1 /etc/crowdsec/config/scenarios/iptables-scan-multi_ports.yaml
crowdsecurity/http-crawl-non_statics ✔️ enabled 0.2 /etc/crowdsec/config/scenarios/http-crawl-non_statics.yaml
crowdsecurity/http-sqli-probing ✔️ enabled 0.2 /etc/crowdsec/config/scenarios/http-sqli-probing.yaml
crowdsecurity/ssh-bf ✔️ enabled 0.1 /etc/crowdsec/config/scenarios/ssh-bf.yaml
crowdsecurity/http-probing ✔️ enabled 0.1 /etc/crowdsec/config/scenarios/http-probing.yaml
crowdsecurity/http-backdoors-attempts ✔️ enabled 0.2 /etc/crowdsec/config/scenarios/http-backdoors-attempts.yaml
crowdsecurity/postfix-spam ✔️ enabled 0.1 /etc/crowdsec/config/scenarios/postfix-spam.yaml
crowdsecurity/http-bad-user-agent ✔️ enabled 0.2 /etc/crowdsec/config/scenarios/http-bad-user-agent.yaml
crowdsecurity/http-path-traversal-probing ✔️ enabled 0.2 /etc/crowdsec/config/scenarios/http-path-traversal-probing.yaml
crowdsecurity/http-xss-probing ✔️ enabled 0.2 /etc/crowdsec/config/scenarios/http-xss-probing.yaml
crowdsecurity/mysql-bf ✔️ enabled 0.1 /etc/crowdsec/config/scenarios/mysql-bf.yaml
----------------------------------------------------------------------------------------------------------------------------------
INFO[0000] COLLECTIONS:
--------------------------------------------------------------------------------------------------------------------
NAME �� STATUS VERSION LOCAL PATH
--------------------------------------------------------------------------------------------------------------------
crowdsecurity/mysql ✔️ enabled 0.1 /etc/crowdsec/config/collections/mysql.yaml
crowdsecurity/linux ✔️ enabled 0.2 /etc/crowdsec/config/collections/linux.yaml
crowdsecurity/nginx ✔️ enabled 0.1 /etc/crowdsec/config/collections/nginx.yaml
crowdsecurity/postfix ✔️ enabled 0.1 /etc/crowdsec/config/collections/postfix.yaml
crowdsecurity/sshd ✔️ enabled 0.1 /etc/crowdsec/config/collections/sshd.yaml
crowdsecurity/apache2 ✔️ enabled 0.1 /etc/crowdsec/config/collections/apache2.yaml
crowdsecurity/base-http-scenarios ✔️ enabled 0.1 /etc/crowdsec/config/collections/base-http-scenarios.yaml
crowdsecurity/iptables ✔️ enabled 0.1 /etc/crowdsec/config/collections/iptables.yaml
--------------------------------------------------------------------------------------------------------------------
INFO[0000] POSTOVERFLOWS:
------------------------------------------------------------------------------------------------------------------------
NAME �� STATUS VERSION LOCAL PATH
------------------------------------------------------------------------------------------------------------------------
crowdsecurity/cdn-whitelist ✔️ enabled 0.2 /etc/crowdsec/config/postoverflows/s01-whitelist/cdn-whitelist.yaml
------------------------------------------------------------------------------------------------------------------------
I am currently look at /var/log/crowdsec.log for errors - no errros found but hereq are a couple of lines that look suspicious (I am referring to the one about “no filename or filenames”
time="02-11-2020 13:39:35" level=warning msg="Loaded 13 scenarios"
time="02-11-2020 13:39:35" level=warning msg="Restoring buckets state from /tmp/crowdsec-buckets-dump-843844857"
time="02-11-2020 13:39:35" level=info msg="Restored 0 buckets from dump"
time="02-11-2020 13:39:35" level=info msg="Loading output profiles"
time="02-11-2020 13:39:35" level=info msg="Loading API client"
time="02-11-2020 13:39:35" level=info msg="API connector init"
time="02-11-2020 13:39:35" level=info msg="api load configuration: configuration loaded successfully (base:https://tmsov6x2n9.execute-api.eu-west-1.amazo
naws.com/v1/)"
time="02-11-2020 13:39:35" level=info msg="api signin: signed in successfuly"
time="02-11-2020 13:39:35" level=info msg="Opening file '/var/log/auth.log' (pattern:/var/log/auth.log)"
time="02-11-2020 13:39:35" level=info msg="Opening file '/var/log/syslog' (pattern:/var/log/syslog)"
time="02-11-2020 13:39:35" level=info msg="Opening file '/var/log/kern.log' (pattern:/var/log/kern.log)"
time="02-11-2020 13:39:35" level=info msg="Opening file '/var/log/messages' (pattern:/var/log/messages)"
time="02-11-2020 13:39:35" level=info msg="No filename or filenames, skipping empty item {Type:file Mode:tail Filename: Filenames:[] tail:<nil> Labels:ma
p[] Profiling:false}"
time="02-11-2020 13:39:35" level=info msg="Starting processing routines"
time="02-11-2020 13:39:35" level=warning msg="Starting processing data"
time="02-11-2020 13:39:35" level=info msg="starting (tail) reader file 0/4 : /var/log/auth.log"
time="02-11-2020 13:39:35" level=info msg="starting (tail) reader file 1/4 : /var/log/syslog"
time="02-11-2020 13:39:35" level=info msg="starting (tail) reader file 2/4 : /var/log/kern.log"
time="02-11-2020 13:39:35" level=info msg="starting (tail) reader file 3/4 : /var/log/messages"
time="02-11-2020 13:39:35" level=info msg="Started 4 routines for polling/read"
time="02-11-2020 13:39:35" level=info msg="Reload is finished"
ovizii
November 6, 2020, 2:18pm
4
on another note, I tried adding the log files for postfix since I had already installed the parsers and scenarios for postfix but even after reloading crowdsec service the metrics don’t reflect it
edited /etc/crowdsec/config/acquis.yaml and added my code below the existing entries:
#Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/auth.log
filenames:
- /var/log/auth.log
labels:
type: syslog
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log /var/log/messages
filenames:
- /var/log/syslog
- /var/log/kern.log
- /var/log/messages
labels:
type: syslog
---
#manually added by ovi
filenames:
- /var/log/mail.log
- /var/log/mail.info
- /var/log/mail.warn
- /var/log/mail.err
labels:
type: syslog
---
systemctl reload crowdsec
INFO[0000] Buckets Metrics:
+--------------------------------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/ssh-bf | - | 1 | 2 | 8 | 1 |
| crowdsecurity/ssh-bf_user-enum | - | - | 1 | 1 | 1 |
+--------------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+-------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log | 4306 | 8 | 4298 | 9 |
| /var/log/kern.log | 931 | - | 931 | - |
| /var/log/messages | 1022 | - | 1022 | - |
| /var/log/syslog | 71290 | - | 71290 | - |
+-------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+--------------------------------+-------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+--------------------------------+-------+--------+----------+
| child-crowdsecurity/sshd-logs | 34 | 8 | 26 |
| crowdsecurity/cdn-whitelist | 3 | 3 | - |
| crowdsecurity/dateparse-enrich | 8 | 8 | - |
| crowdsecurity/geoip-enrich | 8 | 8 | - |
| crowdsecurity/iptables-logs | 2793 | - | 2793 |
| crowdsecurity/sshd-logs | 10 | 8 | 2 |
| crowdsecurity/syslog-logs | 77549 | 77549 | - |
| crowdsecurity/whitelists | 8 | 8 | - |
+--------------------------------+-------+--------+----------+
ErenJ
November 20, 2020, 3:17pm
5
Did you try to reload or restart crowdsec service after adding this configuration ?
ovizii
November 20, 2020, 3:32pm
6
Yes, I mentioned it between those two code blocks above.
Does this look alright?
systemctl status crowdsec
● crowdsec.service - Crowdwatch agent
Loaded: loaded (/etc/systemd/system/crowdsec.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2020-11-20 16:32:40 CET; 26s ago
Process: 27431 ExecStart=/usr/local/bin/crowdsec -c /etc/crowdsec/config/default.yaml (code=exited, status=0/SUCCESS)
Process: 27449 ExecStartPost=/bin/sleep 0.1 (code=exited, status=0/SUCCESS)
Process: 28035 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 27443 (crowdsec)
Tasks: 25 (limit: 4915)
Memory: 46.9M
CGroup: /system.slice/crowdsec.service
└─27443 /usr/local/bin/crowdsec -c /etc/crowdsec/config/default.yaml
Nov 20 16:32:40 nas systemd[1]: Starting Crowdwatch agent...
Nov 20 16:32:40 nas systemd[1]: Started Crowdwatch agent.
ErenJ
November 20, 2020, 5:31pm
7
Sorry didn’t see. Yes looks good.
When you reload crowdsec, you should new in /var/log/crowdsec.log :
time="20-11-2020 00:00:00" level=info msg="Starting tail of /var/log/mail.*"
And metrics about your new filenames (/var/log/mail.*).
1 Like
ovizii
January 21, 2021, 12:05pm
8
hi there, can we give this another try?
I have a brand new Debian buster server where the auth.log files are full of random failed ssh login attempts.
I followed the crowdsec installation instructions for 1.0.4 then installed the cs-firewall-bouncer. Restarted the crowdsec service.
Still seeing plenty of unparsed logs / lines.
Please have a look:
/crowdsec-v1.0.4# cscli metrics
INFO[0000] Buckets Metrics:
+--------------------------------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/ssh-bf | - | 1 | 2 | 9 | 1 |
| crowdsecurity/ssh-bf_user-enum | - | - | 1 | 1 | 1 |
+--------------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+--------------------------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log | 22 | 9 | 13 | 10 |
| /var/log/proxmox-backup/api/auth.log | 501 | - | 501 | - |
| /var/log/syslog | 87 | - | 87 | - |
+--------------------------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+----------------------------------+------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+----------------------------------+------+--------+----------+
| child-crowdsecurity/sshd-logs | 74 | 9 | 65 |
| crowdsecurity/cdn-whitelist | 1 | 1 | - |
| crowdsecurity/dateparse-enrich | 9 | 9 | - |
| crowdsecurity/geoip-enrich | 9 | 9 | - |
| crowdsecurity/rdns | 1 | 1 | - |
| crowdsecurity/seo-bots-whitelist | 1 | 1 | - |
| crowdsecurity/sshd-logs | 22 | 9 | 13 |
| crowdsecurity/syslog-logs | 610 | 109 | 501 |
| crowdsecurity/whitelists | 9 | 9 | - |
+----------------------------------+------+--------+----------+
INFO[0000] Local Api Metrics:
+----------------------+--------+------+
| ROUTE | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts | GET | 3 |
| /v1/alerts | POST | 1 |
| /v1/decisions/stream | GET | 61 |
| /v1/watchers/login | POST | 11 |
+----------------------+--------+------+
INFO[0000] Local Api Machines Metrics:
+----------------------------------+------------+--------+------+
| MACHINE | ROUTE | METHOD | HITS |
+----------------------------------+------------+--------+------+
| e379ff56fb8440bb90e488dcbca79dfc | /v1/alerts | GET | 3 |
| e379ff56fb8440bb90e488dcbca79dfc | /v1/alerts | POST | 1 |
+----------------------------------+------------+--------+------+
INFO[0000] Local Api Bouncers Metrics:
+------------------------------+----------------------+--------+------+
| BOUNCER | ROUTE | METHOD | HITS |
+------------------------------+----------------------+--------+------+
| cs-firewall-bouncer-RpoVeWs2 | /v1/decisions/stream | GET | 61 |
+------------------------------+----------------------+--------+------+
ErenJ
January 21, 2021, 5:25pm
9
Hi @ovizii ,
It seems your parser doing well (it parsed 9 lines and there are 13 others unparsed). Only the essential logs to detect attacks are parsed (sshd parser and patterns ), so it can be possible to have unparsed logs.
If you’re talking about /var/log/proxmox-backup/api/auth.log, it’s another point. Indeed, those logs aren’t known for now and need to write a parser + scenario (contributing doc ) to detect fail authentications.
ErenJ
January 22, 2021, 8:55am
10
ovizii
January 22, 2021, 9:33am
11
Awesome, thanks. I’m glad everything is working.
I have another related question, maybe someone can point me towards the right direction to read up on it.
I have been using fail2ban for years and I am wondering what the “parameters” of crowdsec are i.e. after how may tries in which period does a ban occur? For how long will a ban last? Are these customizable?
ErenJ
January 27, 2021, 10:11am
12
Sorry for the delay.
Those parameters are all customizable, they are managed by what we called a scenario (explained here ).
You can write your own scenarios following this documentation .
1 Like
