Crowdsec container and journald matches issues

Hi all

Testing crowdsec for a new fail2ban replacement in centos9 stream, I cannot have the log parsed with journald acquisition, only with file://var/log/secure

  • using podman in a root container
  • lauching metrics with journald and file acquisition
podman run -d --replace  -v crowdsec_config:/etc/crowdsec   \
-v crowdsec_data:/var/lib/crowdsec/data       -v /run/log/journal/:/run/log/journal:Z   \
-v /var/log/secure:/var/log/secure:Z    -e COLLECTIONS="crowdsecurity/sshd"    \
-p 8080:8080 -p 6060:6060    \
--name crowdsec docker.io/crowdsecurity/crowdsec:latest-debian

in /var/lib/containers/storage/volumes/crowdsec_config/_data/acquis.yaml

---
source: journalctl
journalctl_filter:
 - "_SYSTEMD_UNIT=sshd.service"
labels:
  type: journald
---
filenames:
- /var/log/secure
labels:
  type: syslog
---

I can see decisions only after I set the /var/log/secure in acquisition, when I watch the metcrics I have

Every 2.0s: podman exec -ti crowdsec cscli metrics                                                                                                                                            droplet1.nethserver.fr: Mon Sep 12 10:30:27 2022

^[36mINFO^[0m[12-09-2022 02:30:28 PM] Buckets Metrics:                             g
+-------------------------------------+---------------+-----------+--------------+--------+---------+-
|               BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANTIATED | POURED | EXPIRED |E
+-------------------------------------+---------------+-----------+--------------+--------+---------+-
| crowdsecurity/ssh-bf                | 1             | -         | 3            | 10     | 2       |
| crowdsecurity/ssh-bf_user-enum      | 1             | -         | 3            | 5      | 2       |-
| crowdsecurity/ssh-slow-bf           | 3             | -         | 3            | 10	  | -       |
| crowdsecurity/ssh-slow-bf_user-enum | 3             | -         | 3            | 5	  | -       |
+-------------------------------------+---------------+-----------+--------------+--------+---------+
^[36mINFO^[0m[12-09-2022 02:30:28 PM] Acquisition Metrics:
+--------------------------------------------------+------------+--------------+----------------+------------------------+
|                      SOURCE                      | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/secure                             | 16         | 10           | 6              | 30                     |
| journalctl:journalctl-_SYSTEMD_UNIT=sshd.service | 15         | -            | 15             | -                      |
+--------------------------------------------------+------------+--------------+----------------+------------------------+
^[36mINFO^[0m[12-09-2022 02:30:28 PM] Parser Metrics:
+---------------------------------+------+--------+----------+
|             PARSERS             | HITS | PARSED | UNPARSED |
+---------------------------------+------+--------+----------+
| child-crowdsecurity/sshd-logs   | 110  | 10	  | 100      |
| child-crowdsecurity/syslog-logs | 16   | 16	  | -        |
| crowdsecurity/dateparse-enrich  | 10   | 10	  | -        |
| crowdsecurity/geoip-enrich	  | 10   | 10	  | -        |
| crowdsecurity/non-syslog        | 15   | 15	  | -        |
| crowdsecurity/sshd-logs         | 15   | 10	  | 5        |
| crowdsecurity/syslog-logs	  | 16   | 16	  | -        |
| crowdsecurity/whitelists        | 10   | 10     | -        |
+---------------------------------+------+--------+----------+
^[36mINFO^[0m[12-09-2022 02:30:28 PM] Local Api Metrics:
+--------------------+--------+------+
|	ROUTE        | METHOD | HITS |
+--------------------+--------+------+
| /v1/heartbeat      | GET    | 5    |
| /v1/watchers/login | POST   | 2    |
+--------------------+--------+------+L
^[36mINFO^[0m[12-09-2022 02:30:28 PM] Local Api Machines Metrics:
+-----------+---------------+--------+------+O
|  MACHINE  |     ROUTE     | METHOD | HITS |-
+-----------+---------------+--------+------+
| localhost | /v1/heartbeat | GET    | 5    |-
+-----------+---------------+--------+------+p
^[36mINFO^[0m[12-09-2022 02:30:28 PM] Local Api Decisions:
+---------------------------+----------+--------+-------+
|          REASON           |  ORIGIN  | ACTION | COUNT |
+---------------------------+----------+--------+-------+
| crowdsecurity/ssh-slow-bf | crowdsec | ban    | 9     |
+---------------------------+----------+--------+-------+
^[36mINFO^[0m[12-09-2022 02:30:28 PM] Local Api Alerts:
+---------------------------+-------+
|          REASON           | COUNT |
+---------------------------+-------+
| crowdsecurity/ssh-slow-bf | 9     |

however the debian container seems to see well the journal

root@f164a0edaf92:/# journalctl -f -u sshd.service
-- Journal begins at Mon 2022-09-12 13:04:28 UTC. --
Sep 12 14:26:05 droplet1.nethserver.fr sshd[98228]: Invalid user admin from 89.252.140.21 port 44938
Sep 12 14:26:05 droplet1.nethserver.fr sshd[98228]: pam_unix(sshd:auth): check pass; user unknown
Sep 12 14:26:05 droplet1.nethserver.fr sshd[98228]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=89.252.140.21
Sep 12 14:26:07 droplet1.nethserver.fr sshd[98228]: Failed password for invalid user admin from 89.252.140.21 port 44938 ssh2
Sep 12 14:26:08 droplet1.nethserver.fr sshd[98228]: Received disconnect from 89.252.140.21 port 44938:11: Bye Bye [preauth]
Sep 12 14:26:08 droplet1.nethserver.fr sshd[98228]: Disconnected from invalid user admin 89.252.140.21 port 44938 [preauth]
Sep 12 14:29:37 droplet1.nethserver.fr sshd[100707]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=92.255.85.70  user=root
Sep 12 14:29:39 droplet1.nethserver.fr sshd[100707]: Failed password for root from 92.255.85.70 port 61768 ssh2
Sep 12 14:29:39 droplet1.nethserver.fr sshd[100707]: Received disconnect from 92.255.85.70 port 61768:11: Bye Bye [preauth]
Sep 12 14:29:39 droplet1.nethserver.fr sshd[100707]: Disconnected from authenticating user root 92.255.85.70 port 61768 [preauth]

If I remove the /var/log/secure acquisition and restart crowdsec I can see something changed in the metrics

Every 2.0s: podman exec -ti crowdsec cscli metrics                                                                                                                                            droplet1.nethserver.fr: Mon Sep 12 10:37:32 2022

^[36mINFO^[0m[12-09-2022 02:37:33 PM] Acquisition Metrics:                         g
+--------------------------------------------------+------------+--------------+----------------+------------------------+
|                      SOURCE                      | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------------------------+------------+--------------+----------------+------------------------+
| journalctl:journalctl-_SYSTEMD_UNIT=sshd.service | 1          | -            | 1              | -                      |
+--------------------------------------------------+------------+--------------+----------------+------------------------+
^[36mINFO^[0m[12-09-2022 02:37:33 PM] Parser Metrics:
+--------------------------+------+--------+----------+
|         PARSERS          | HITS | PARSED | UNPARSED |
+--------------------------+------+--------+----------+
| crowdsecurity/non-syslog | 1    | 1	   | -        |
+--------------------------+------+--------+----------+
^[36mINFO^[0m[12-09-2022 02:37:33 PM] Local Api Metrics:
+--------------------+--------+------+
|	ROUTE        | METHOD | HITS |
+--------------------+--------+------+
| /v1/heartbeat      | GET    | 2    |
| /v1/watchers/login | POST   | 2    |
+--------------------+--------+------+L
^[36mINFO^[0m[12-09-2022 02:37:33 PM] Local Api Machines Metrics:
+-----------+---------------+--------+------+O
|  MACHINE  |     ROUTE     | METHOD | HITS |-
+-----------+---------------+--------+------+
| localhost | /v1/heartbeat | GET    | 2    |-
+-----------+---------------+--------+------+p
^[36mINFO^[0m[12-09-2022 02:37:33 PM] Local Api Decisions:
+---------------------------+----------+--------+-------+
|          REASON           |  ORIGIN  | ACTION | COUNT |
+---------------------------+----------+--------+-------+
| crowdsecurity/ssh-slow-bf | crowdsec | ban    | 9     |
+---------------------------+----------+--------+-------+
^[36mINFO^[0m[12-09-2022 02:37:33 PM] Local Api Alerts:
+---------------------------+-------+
|          REASON           | COUNT |
+---------------------------+-------+
| crowdsecurity/ssh-slow-bf | 9     |
+---------------------------+-------+

I miss all the buckets, not sure it is wanted ???

And all the parsers ???

answering myself I suppose the documentation send to a wrong direction, the doc state for version 1.41

---
source: journalctl
journalctl_filter:
  - "_SYSTEMD_UNIT=sshd.service"
labels:
  type: journald
---

but it does not work

however for the version 1.0 the documentation states

---
source: journalctl
journalctl_filter:
  - "_SYSTEMD_UNIT=sshd.service"
labels:
  type: syslog
---

yaml filenames: - /var/log/nginx/access*.log - /var/log/nginx/error.log labels: type: nginx --- filenames: - /var/log/auth.log labels: type: syslog --- journalctl_filter: - "_SYSTEMD_UNIT=ssh.service" labels: type: syslog

and this syntax is good and workable now

[root@droplet1 ~]# podman exec -ti crowdsec cscli metrics
INFO[12-09-2022 05:15:28 PM] Buckets Metrics:                             
+-------------------------------------+---------------+-----------+--------------+--------+---------+
|               BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANTIATED | POURED | EXPIRED |
+-------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/ssh-bf                | -             | -         | 4            | 19     | 4       |
| crowdsecurity/ssh-bf_user-enum      | -             | -         | 4            | 7      | 4       |
| crowdsecurity/ssh-slow-bf           | 3             | 1         | 4            | 19     | -       |
| crowdsecurity/ssh-slow-bf_user-enum | 3             | -         | 3            | 5      | -       |
+-------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[12-09-2022 05:15:28 PM] Acquisition Metrics:                         
+--------------------------------------------------+------------+--------------+----------------+------------------------+
|                      SOURCE                      | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------------------------+------------+--------------+----------------+------------------------+
| journalctl:journalctl-_SYSTEMD_UNIT=sshd.service | 39         | 19           | 20             | 50                     |
+--------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[12-09-2022 05:15:28 PM] Parser Metrics:                              
+---------------------------------+------+--------+----------+
|             PARSERS             | HITS | PARSED | UNPARSED |
+---------------------------------+------+--------+----------+
| child-crowdsecurity/sshd-logs   | 315  | 19     | 296      |
| child-crowdsecurity/syslog-logs | 40   | 38     | 2        |
| crowdsecurity/dateparse-enrich  | 19   | 19     | -        |
| crowdsecurity/geoip-enrich      | 19   | 19     | -        |
| crowdsecurity/sshd-logs         | 37   | 19     | 18       |
| crowdsecurity/syslog-logs       | 39   | 38     | 1        |
| crowdsecurity/whitelists        | 19   | 19     | -        |
+---------------------------------+------+--------+----------+
INFO[12-09-2022 05:15:28 PM] Local Api Metrics:                           
+--------------------+--------+------+
|       ROUTE        | METHOD | HITS |
+--------------------+--------+------+
| /manager/text/list | GET    | 1    |
| /v1/alerts         | GET    | 5    |
| /v1/alerts         | POST   | 1    |
| /v1/heartbeat      | GET    | 9    |
| /v1/watchers/login | POST   | 7    |
+--------------------+--------+------+
INFO[12-09-2022 05:15:28 PM] Local Api Machines Metrics:                  
+-----------+---------------+--------+------+
|  MACHINE  |     ROUTE     | METHOD | HITS |
+-----------+---------------+--------+------+
| localhost | /v1/alerts    | GET    | 5    |
| localhost | /v1/alerts    | POST   | 1    |
| localhost | /v1/heartbeat | GET    | 9    |
+-----------+---------------+--------+------+
INFO[12-09-2022 05:15:28 PM] Local Api Decisions:                         
+--------------------------------------------+----------+--------+-------+
|                   REASON                   |  ORIGIN  | ACTION | COUNT |
+--------------------------------------------+----------+--------+-------+
| crowdsecurity/http-probing                 | CAPI     | ban    | 434   |
| crowdsecurity/jira_cve-2021-26086          | CAPI     | ban    | 25    |
| crowdsecurity/ssh-bf                       | CAPI     | ban    | 254   |
| crowdsecurity/fortinet-cve-2018-13379      | CAPI     | ban    | 41    |
| crowdsecurity/http-backdoors-attempts      | CAPI     | ban    | 44    |
| crowdsecurity/http-crawl-non_statics       | CAPI     | ban    | 1692  |
| crowdsecurity/http-cve-2021-41773          | CAPI     | ban    | 4     |
| crowdsecurity/http-open-proxy              | CAPI     | ban    | 76    |
| crowdsecurity/apache_log4j2_cve-2021-44228 | CAPI     | ban    | 15    |
| crowdsecurity/f5-big-ip-cve-2020-5902      | CAPI     | ban    | 2     |
| crowdsecurity/http-bad-user-agent          | CAPI     | ban    | 2416  |
| crowdsecurity/http-sensitive-files         | CAPI     | ban    | 96    |
| crowdsecurity/thinkphp-cve-2018-20062      | CAPI     | ban    | 163   |
| crowdsecurity/http-generic-bf              | CAPI     | ban    | 16    |
| crowdsecurity/http-path-traversal-probing  | CAPI     | ban    | 51    |
| crowdsecurity/ssh-slow-bf                  | crowdsec | ban    | 10    |
| crowdsecurity/ssh-slow-bf                  | CAPI     | ban    | 7500  |
| ltsich/http-w00tw00t                       | CAPI     | ban    | 6     |
+--------------------------------------------+----------+--------+-------+
INFO[12-09-2022 05:15:28 PM] Local Api Alerts:                            
+---------------------------+-------+
|          REASON           | COUNT |
+---------------------------+-------+
| crowdsecurity/ssh-slow-bf | 10    |
+---------------------------+-------+

cc @klausagnoletti @thibault do I am right ?

I run the latest crowdsec debian container : docker.io/crowdsecurity/crowdsec:latest-debian

answer from the developers in the discord channel, the doc is wrong, use type: syslog and not type: journald