I’ve gone through all the unparsed issues noted in this forum, but nothing so far has fixed an issue which I’m particularly concerned about. VSFTPD is not being parsed. Of course, I’ve installed gettext and restarted crowdsec service. It’s been running for a few weeks now. Not sure if it’s a permissions thing or what. Any help would be appreciated. Release: 1.0.4
Hi @cavaughan,
Could you please type cscli metrics and paste here the result.
INFO[0000] Buckets Metrics:
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-bad-user-agent | - | 3 | 79 | 83 | 76 |
| crowdsecurity/http-crawl-non_statics | 1 | - | 705 | 738 | 704 |
| crowdsecurity/http-path-traversal-probing | - | - | 3 | 3 | 3 |
| crowdsecurity/http-probing | - | - | 183 | 190 | 183 |
| crowdsecurity/http-sensitive-files | - | - | 4 | 4 | 4 |
+-------------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+-----------------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------+------------+--------------+----------------+------------------------+
| /var/log/apache2/access.log | 1022 | 1022 | - | 1018 |
| /var/log/apache2/error.log | 24 | - | 24 | - |
| /var/log/auth.log | 9839 | - | 9839 | - |
| /var/log/kern.log | 146 | - | 146 | - |
| /var/log/syslog | 31369 | - | 31369 | - |
+-----------------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+----------------------------------+-------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+----------------------------------+-------+--------+----------+
| child-crowdsecurity/apache2-logs | 1070 | 1022 | 48 |
| child-crowdsecurity/http-logs | 3066 | 1462 | 1604 |
| child-crowdsecurity/sshd-logs | 1025 | - | 1025 |
| crowdsecurity/apache2-logs | 1046 | 1022 | 24 |
| crowdsecurity/cdn-whitelist | 3 | 3 | - |
| crowdsecurity/dateparse-enrich | 1022 | 1022 | - |
| crowdsecurity/geoip-enrich | 1022 | 1022 | - |
| crowdsecurity/http-logs | 1022 | 417 | 605 |
| crowdsecurity/iptables-logs | 293 | - | 293 |
| crowdsecurity/non-syslog | 1046 | 1046 | - |
| crowdsecurity/rdns | 3 | 3 | - |
| crowdsecurity/seo-bots-whitelist | 3 | 3 | - |
| crowdsecurity/sshd-logs | 205 | - | 205 |
| crowdsecurity/syslog-logs | 41354 | 41354 | - |
| crowdsecurity/whitelists | 1022 | 1022 | - |
| vsftpd-logs | 916 | - | 916 |
+----------------------------------+-------+--------+----------+
INFO[0000] Local Api Metrics:
+--------------------+--------+------+
| ROUTE | METHOD | HITS |
+--------------------+--------+------+
| /v1/alerts | GET | 5 |
| /v1/alerts | POST | 3 |
| /v1/alerts/146 | GET | 1 |
| /v1/watchers/login | POST | 11 |
+--------------------+--------+------+
INFO[0000] Local Api Machines Metrics:
+----------------------------------+----------------+--------+------+
| MACHINE | ROUTE | METHOD | HITS |
+----------------------------------+----------------+--------+------+
| 173ac883192e107fc2f5f1e6586d8251 | /v1/alerts | POST | 3 |
| 173ac883192e107fc2f5f1e6586d8251 | /v1/alerts | GET | 5 |
| 173ac883192e107fc2f5f1e6586d8251 | /v1/alerts/146 | GET | 1 |
+----------------------------------+----------------+--------+------+
Hello,
From your config / metrics, it seems that you have a “custom” vsftpd log parser, and that it doesn’t succeed to parse any of the 916 lines of logs that were parsed (see the line with vsftpd-logs in your parser metrics).
Would you mind sharing some logs and/or the parser so we can see what’s going on ? 
Cheers,
Sure, but which logs do you want me to post. Seems like just posting vsftp logs wouldn’t provide any pertinent info. Also I definitely don’t want to post my vsftp logs here.
As a parser… not sure what you want me to share…
Hello,
What I meant is that, given that there are already some tests for vsftpd logs, this parser is assumed to work. Given this, there are at least two suspects :
- [1] you modified your parser (and doesn’t work anymore ?)
- [2] your logs are not standard
To check [1] maybe you should check the output of cscli parsers list : it will tell you if the parser is up-to-date (but I guess not) and will point you to the path of the parser’s yaml configuration file.
For [2], if you don’t want to share a log sample (that would allow to ensure that your logs are parsed by the upstream parser), you should check your config to tell if you’re using a custom format ?
I would advise trying to install the upstream parser if yours differs. You can find it here if you want to check the diffs with yours.
I have no problem sending the ftp logs to you directly, but I don’t want to publish them on a public server.
Testing to see if responding directly to your email goes anywhere.
cscli parsers list
I have the same problem. vsftpd-logs metrics shows all Unparsed. But explain shows that everything is ok. Logs are in this format (ip changed):
Sun Oct 13 00:16:52 2024 [pid 1267556] CONNECT: Client "1.2.3.4"
Sun Oct 13 00:16:54 2024 [pid 1267555] [anonymous] FAIL LOGIN: Client "1.2.3.4"
explain
line: Sun Oct 13 00:16:52 2024 [pid 1267556] CONNECT: Client "1.2.3.4"
├ s00-raw
| ├ 🔴 crowdsecurity/syslog-logs
| └ 🟢 crowdsecurity/non-syslog (+5 ~8)
├ s01-parse
| ├ 🔴 crowdsecurity/mysql-logs
| ├ 🔴 crowdsecurity/nginx-logs
| ├ 🔴 crowdsecurity/smb-logs
| ├ 🔴 crowdsecurity/sshd-logs
| └ 🔴 vsftpd-logs
└-------- parser failure 🔴
line: Sun Oct 13 00:16:54 2024 [pid 1267555] [anonymous] FAIL LOGIN: Client "1.2.3.4"
├ s00-raw
| ├ 🔴 crowdsecurity/syslog-logs
| └ 🟢 crowdsecurity/non-syslog (+5 ~8)
├ s01-parse
| ├ 🔴 crowdsecurity/mysql-logs
| ├ 🔴 crowdsecurity/nginx-logs
| ├ 🔴 crowdsecurity/smb-logs
| ├ 🔴 crowdsecurity/sshd-logs
| └ 🟢 vsftpd-logs (+7 ~3)
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
| ├ 🟢 crowdsecurity/geoip-enrich (+10)
| ├ 🔴 crowdsecurity/http-logs
| └ 🟢 crowdsecurity/whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
└ 🟢 crowdsecurity/vsftpd-bf
metrics
Acquisition Metrics:
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
| Source | Lines read | Lines parsed | Lines unparsed | Lines poured to bucket | Lines whitelisted |
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
| file:/var/log/auth.log | 114.43k | 28.82k | 85.60k | 69.92k | 8 |
| file:/var/log/samba/samba.log | 492.10M | 5.29M | 486.81M | - | 5.29M |
| file:/var/log/syslog | 68.75k | - | 68.75k | - | - |
| journalctl:journalctl-_SYSTEMD_UNIT=nginx.service | 7 | - | 7 | - | - |
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
Local API Alerts:
+-------------------------------------+-------+
| Reason | Count |
+-------------------------------------+-------+
| crowdsecurity/ssh-bf | 1530 |
| crowdsecurity/ssh-bf_user-enum | 121 |
| crowdsecurity/ssh-slow-bf | 3155 |
| crowdsecurity/ssh-slow-bf_user-enum | 116 |
+-------------------------------------+-------+
Local API Decisions:
+--------------------------------------------+----------+--------+-------+
| Reason | Origin | Action | Count |
+--------------------------------------------+----------+--------+-------+
| crowdsecurity/CVE-2022-35914 | CAPI | ban | 7 |
| crowdsecurity/CVE-2023-49103 | CAPI | ban | 37 |
| crowdsecurity/apache_log4j2_cve-2021-44228 | CAPI | ban | 56 |
| crowdsecurity/http-open-proxy | CAPI | ban | 2861 |
| crowdsecurity/smb-bf | CAPI | ban | 1262 |
| crowdsecurity/http-bad-user-agent | CAPI | ban | 19123 |
| crowdsecurity/http-probing | CAPI | ban | 6980 |
| crowdsecurity/CVE-2017-9841 | CAPI | ban | 441 |
| crowdsecurity/http-admin-interface-probing | CAPI | ban | 393 |
| crowdsecurity/ssh-slow-bf | CAPI | ban | 7044 |
| crowdsecurity/ssh-slow-bf | crowdsec | ban | 2 |
| ltsich/http-w00tw00t | CAPI | ban | 4 |
| crowdsecurity/CVE-2022-26134 | CAPI | ban | 9 |
| crowdsecurity/http-generic-bf | CAPI | ban | 45 |
| crowdsecurity/f5-big-ip-cve-2020-5902 | CAPI | ban | 1 |
| crowdsecurity/http-backdoors-attempts | CAPI | ban | 211 |
| crowdsecurity/http-crawl-non_statics | CAPI | ban | 562 |
| crowdsecurity/http-sensitive-files | CAPI | ban | 498 |
| crowdsecurity/mysql-bf | CAPI | ban | 39 |
| crowdsecurity/CVE-2022-37042 | CAPI | ban | 2 |
| crowdsecurity/fortinet-cve-2018-13379 | CAPI | ban | 15 |
| crowdsecurity/http-cve-2021-41773 | CAPI | ban | 552 |
| crowdsecurity/http-cve-2021-42013 | CAPI | ban | 7 |
| crowdsecurity/http-path-traversal-probing | CAPI | ban | 277 |
| crowdsecurity/jira_cve-2021-26086 | CAPI | ban | 21 |
| crowdsecurity/netgear_rce | CAPI | ban | 135 |
| crowdsecurity/CVE-2023-22515 | CAPI | ban | 6 |
| crowdsecurity/http-cve-probing | CAPI | ban | 53 |
| crowdsecurity/nginx-req-limit-exceeded | CAPI | ban | 772 |
| crowdsecurity/ssh-cve-2024-6387 | CAPI | ban | 52 |
| crowdsecurity/thinkphp-cve-2018-20062 | CAPI | ban | 210 |
| crowdsecurity/vsftpd-bf | CAPI | ban | 2 |
| crowdsecurity/CVE-2019-18935 | CAPI | ban | 43 |
| crowdsecurity/http-wordpress-scan | CAPI | ban | 814 |
| crowdsecurity/spring4shell_cve-2022-22965 | CAPI | ban | 1 |
| crowdsecurity/ssh-bf | crowdsec | ban | 1 |
| crowdsecurity/ssh-bf | CAPI | ban | 6746 |
+--------------------------------------------+----------+--------+-------+
Local API Metrics:
+----------------------+--------+-------+
| Route | Method | Hits |
+----------------------+--------+-------+
| /v1/alerts | GET | 9 |
| /v1/alerts | POST | 2762 |
| /v1/decisions | DELETE | 1 |
| /v1/decisions/stream | GET | 35786 |
| /v1/heartbeat | GET | 26205 |
| /v1/usage-metrics | POST | 668 |
| /v1/watchers/login | POST | 504 |
+----------------------+--------+-------+
Local API Bouncers Metrics:
+---------------+----------------------+--------+-------+
| Bouncer | Route | Method | Hits |
+---------------+----------------------+--------+-------+
| turris_remote | /v1/decisions/stream | GET | 35786 |
+---------------+----------------------+--------+-------+
Local API Machines Metrics:
+--------------------------------------------------+---------------+--------+------+
| Machine | Route | Method | Hits |
+--------------------------------------------------+---------------+--------+------+
| 5df0698592c7488b95d3fd0fd1aa5aecWzmFjfyVQ6SRcj19 | /v1/decisions | DELETE | 1 |
| 5df0698592c7488b95d3fd0fd1aa5aecWzmFjfyVQ6SRcj19 | /v1/alerts | GET | 9 |
| 5df0698592c7488b95d3fd0fd1aa5aecWzmFjfyVQ6SRcj19 | /v1/alerts | POST | 1653 |
| 5df0698592c7488b95d3fd0fd1aa5aecWzmFjfyVQ6SRcj19 | /v1/heartbeat | GET | 8732 |
| 5df0698592c7488b95d3fd0fd1aa5aecyxIW2oHu5vz20L1S | /v1/heartbeat | GET | 8724 |
| 6d85d562d2144efdb1b8aeeeb3783437cwU21Bp3EeBSAWM0 | /v1/heartbeat | GET | 8735 |
| 6d85d562d2144efdb1b8aeeeb3783437cwU21Bp3EeBSAWM0 | /v1/alerts | POST | 1109 |
+--------------------------------------------------+---------------+--------+------+
Parser Metrics:
+---------------------------------+---------+---------+----------+
| Parsers | Hits | Parsed | Unparsed |
+---------------------------------+---------+---------+----------+
| child-crowdsecurity/nginx-logs | 21 | - | 21 |
| child-crowdsecurity/smb-logs | 984.20M | 5.29M | 978.91M |
| child-crowdsecurity/sshd-logs | 649.02k | 28.82k | 620.20k |
| child-crowdsecurity/syslog-logs | 183.18k | 183.18k | - |
| child-vsftpd-logs | 7.18k | - | 7.18k |
| crowdsecurity/dateparse-enrich | 28.82k | 28.82k | - |
| crowdsecurity/geoip-enrich | 28.81k | 28.81k | - |
| crowdsecurity/nginx-logs | 7 | - | 7 |
| crowdsecurity/non-syslog | 492.10M | 492.10M | - |
| crowdsecurity/smb-logs | 492.10M | 5.29M | 486.81M |
| crowdsecurity/sshd-logs | 62.70k | 28.82k | 33.88k |
| crowdsecurity/syslog-logs | 183.18k | 183.18k | - |
| crowdsecurity/whitelists | 5.32M | 5.32M | - |
| vsftpd-logs | 3.59k | - | 3.59k |
+---------------------------------+---------+---------+----------+
Scenario Metrics:
+-------------------------------------+---------------+-----------+--------------+--------+---------+
| Scenario | Current Count | Overflows | Instantiated | Poured | Expired |
+-------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/ssh-bf | - | 1.89k | 3.89k | 28.24k | 2.00k |
| crowdsecurity/ssh-bf_user-enum | - | 264 | 3.09k | 7.92k | 2.83k |
| crowdsecurity/ssh-slow-bf | 1 | 1.79k | 2.49k | 28.24k | 695 |
| crowdsecurity/ssh-slow-bf_user-enum | 1 | 159 | 1.26k | 5.51k | 1.10k |
+-------------------------------------+---------------+-----------+--------------+--------+---------+
Whitelist Metrics:
+--------------------------+-----------------------------+---------+-------------+
| Whitelist | Reason | Hits | Whitelisted |
+--------------------------+-----------------------------+---------+-------------+
| crowdsecurity/whitelists | private ipv4/ipv6 ip/ranges | 5322048 | 5293235 |
+--------------------------+-----------------------------+---------+-------------+
Normally this happens when you type you passed to cscli explain is correct but the one configured in acquisition is not the same. Could you share your acquisition?
before
Acquisition Metrics:
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
| Source | Lines read | Lines parsed | Lines unparsed | Lines poured to bucket | Lines whitelisted |
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
| file:/var/log/auth.log | 128.30k | 29.71k | 98.59k | 72.29k | 9 |
| file:/var/log/samba/samba.log | 669.46M | 7.46M | 662.00M | - | 7.46M |
| file:/var/log/syslog | 80.47k | - | 80.47k | - | - |
| journalctl:journalctl-_SYSTEMD_UNIT=nginx.service | 7 | - | 7 | - | - |
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
I added this to my /etc/crowdsec/acquis.yaml
filenames:
- /var/log/vsftpd.log
labels:
type: vsftpd
---
after
Acquisition Metrics:
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
| Source | Lines read | Lines parsed | Lines unparsed | Lines poured to bucket | Lines whitelisted |
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
| file:/var/log/auth.log | 128.40k | 29.71k | 98.69k | 72.29k | 9 |
| file:/var/log/samba/samba.log | 670.16M | 7.46M | 662.70M | - | 7.46M |
| file:/var/log/syslog | 80.51k | - | 80.51k | - | - |
| file:/var/log/vsftpd.log | 1 | - | 1 | - | - |
| journalctl:journalctl-_SYSTEMD_UNIT=nginx.service | 8 | - | 8 | - | - |
+---------------------------------------------------+------------+--------------+----------------+------------------------+-------------------+
Now it’s working. Thank you.
vsftpd is not supported by wizard.sh?
Nope we havent touched wizard.sh in a while its outdated and we had a project to have cscli setup to take over and allow to detect all services we support, however, development is taking long than expected.