Exec: "journalctl": executable file not found in $PATH

Hi,
Upon installing crowdsec (as a podman quadlet, see below), I wanted it to parse my journald logs.
I installed the crowdsecurity/linux collection but Crowdsec complains about datasource 'journalctl' is not available: exec: \"journalctl\": executable file not found in $PATH".
I found Journalctl parsers fails (solved) but it didn’t help as OP needed to install crowdsecurity/linux.

The quadlet is defined as such:

[Unit]
Description=Crowdsec Quadlet

[Container]
Image=docker.io/crowdsecurity/crowdsec:v1.6.10
ContainerName=crowdsec
Network=tunnel.network
NoNewPrivileges=true
DropCapability=ALL
PublishPort=127.0.0.1:8080:8080
Environment="TZ=Europe/Paris"
Environment="BOUNCER_KEY_caddy={{ $CROWDSEC_CADDY_BOUNCER_API_KEY }}"
Environment="BOUNCER_KEY_firewall={{ $CROWDSEC_FIREWALL_BOUNCER_API_KEY }}"
# Environment="ENROLL_KEY={{ $CROWDSEC_API_KEY }}"
Environment="USE_WAL=true"
Environment="ENROLL_INSTANCE_NAME=telstar"
Environment="COLLECTIONS=crowdsecurity/linux crowdsecurity/sshd crowdsecurity/caddy crowdsecurity/http-cve crowdsecurity/whitelist-good-actors crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules"
# TODO: Dunno if needed, instance is stored in config
Volume=/srv/vic1707/containers/crowdsec/data:/var/lib/crowdsec/data:Z,rw
# TODO: Dunno if needed, custom config is enough?
Volume=/srv/vic1707/containers/crowdsec/config:/etc/crowdsec:Z,rw
Volume=/home/vic1707/.config/containers/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml:Z,rw
Volume=CrowdsecLogs.volume:/var/log/mounted:ro

GroupAdd=keep-groups # Enables read on journal files
# Enables journal mount # TODO: Remove
SecurityLabelDisable=true
Volume=/var/log/journal:/var/log/host:ro

[Service]
Restart=always
ExecStartPre=/usr/bin/bash -c "until [ -f /run/restore-complete ]; do sleep 1; done"
ExecStartPre=/usr/bin/bash -c "until [ -f /run/user/1001/crowdsec-volume-ready ]; do sleep 1; done"

[Install]
WantedBy=default.target

Here’s the acquis.yaml:

---
listen_addr: 127.0.0.1:7422
appsec_config: crowdsecurity/appsec-default
name: CrowdsecAppSec
source: appsec
labels:
  type: appsec

---
source: file
force_inotify: true
filenames:
  - /var/log/mounted/caddy/*.log
labels:
  type: caddy

---
source: journalctl
force_inotify: true
journalctl_filter:
  - "_TRANSPORT=journal"
labels:
  type: syslog

---
source: journalctl
force_inotify: true
journalctl_filter:
  - "_TRANSPORT=syslog"
labels:
  type: syslog

---
source: journalctl
force_inotify: true
journalctl_filter:
  - "_TRANSPORT=stdout"
labels:
  type: syslog

---
source: journalctl
force_inotify: true
journalctl_filter:
  - "_TRANSPORT=kernel"
labels:
  type: syslog

---

Do you know what I’m missing ?

Debug infos:
If anything is missing feel free to ask !

[vic1707@localhost ~]$ podman exec crowdsec cscli metrics
+----------------------------------------------------------------------------------------------------------------------------------+
| Acquisition Metrics                                                                                                              |
+----------------------------------------+------------+--------------+----------------+------------------------+-------------------+
| Source                                 | Lines read | Lines parsed | Lines unparsed | Lines poured to bucket | Lines whitelisted |
+----------------------------------------+------------+--------------+----------------+------------------------+-------------------+
| file:/var/log/mounted/caddy/access.log | 1          | 1            | -              | -                      | 1                 |
+----------------------------------------+------------+--------------+----------------+------------------------+-------------------+
+------------------------------------------------------------------------------------+
| Bouncer Metrics (firewall) since 2025-07-25 21:13:22 +0000 UTC                     |
+----------------------------+------------------+-----------------+------------------+
| Origin                     | active_decisions |     dropped     |     processed    |
|                            |        IPs       | bytes | packets |  bytes | packets |
+----------------------------+------------------+-------+---------+--------+---------+
| CAPI (community blocklist) |           14.00k |     0 |       0 |      - |       - |
+----------------------------+------------------+-------+---------+--------+---------+
|                      Total |           14.00k |     0 |       0 | 42.13M |   9.26k |
+----------------------------+------------------+-------+---------+--------+---------+
+-------------------------------------------+
| Local API Decisions                       |
+-----------------+--------+--------+-------+
| Reason          | Origin | Action | Count |
+-----------------+--------+--------+-------+
| http:crawl      | CAPI   | ban    | 21    |
| http:exploit    | CAPI   | ban    | 8322  |
| http:scan       | CAPI   | ban    | 3244  |
| ssh:bruteforce  | CAPI   | ban    | 2331  |
| ssh:exploit     | CAPI   | ban    | 551   |
| generic:scan    | CAPI   | ban    | 141   |
| http:bruteforce | CAPI   | ban    | 392   |
+-----------------+--------+--------+-------+
+--------------------------------------+
| Local API Metrics                    |
+----------------------+--------+------+
| Route                | Method | Hits |
+----------------------+--------+------+
| /v1/allowlists       | GET    | 230  |
| /v1/decisions/stream | GET    | 1383 |
| /v1/heartbeat        | GET    | 229  |
| /v1/usage-metrics    | POST   | 79   |
| /v1/watchers/login   | POST   | 14   |
+----------------------+--------+------+
+-------------------------------------------------+
| Local API Bouncers Metrics                      |
+----------+----------------------+--------+------+
| Bouncer  | Route                | Method | Hits |
+----------+----------------------+--------+------+
| caddy    | /v1/decisions/stream | GET    | 228  |
| firewall | /v1/decisions/stream | GET    | 1155 |
+----------+----------------------+--------+------+
+--------------------------------------------+
| Local API Machines Metrics                 |
+-----------+----------------+--------+------+
| Machine   | Route          | Method | Hits |
+-----------+----------------+--------+------+
| localhost | /v1/allowlists | GET    | 230  |
| localhost | /v1/heartbeat  | GET    | 229  |
+-----------+----------------+--------+------+
+---------------------------------------------------------------+
| Parser Metrics                                                |
+------------------------------------+------+--------+----------+
| Parsers                            | Hits | Parsed | Unparsed |
+------------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs      | 3    | 2      | 1        |
| crowdsecurity/caddy-logs           | 1    | 1      | -        |
| crowdsecurity/dateparse-enrich     | 1    | 1      | -        |
| crowdsecurity/http-logs            | 1    | 1      | -        |
| crowdsecurity/non-syslog           | 1    | 1      | -        |
| crowdsecurity/public-dns-allowlist | 1    | 1      | -        |
| crowdsecurity/whitelists           | 1    | 1      | -        |
+------------------------------------+------+--------+----------+
+---------------------------------------------------------------------------------------+
| Whitelist Metrics                                                                     |
+------------------------------------+-----------------------------+------+-------------+
| Whitelist                          | Reason                      | Hits | Whitelisted |
+------------------------------------+-----------------------------+------+-------------+
| crowdsecurity/public-dns-allowlist | public DNS server           | 1    | -           |
| crowdsecurity/whitelists           | private ipv4/ipv6 ip/ranges | 1    | 1           |
+------------------------------------+-----------------------------+------+-------------+
[vic1707@localhost ~]$ podman logs crowdsec 2>&1 | grep journalctl
time="2025-07-25T22:58:30+02:00" level=error msg="datasource 'journalctl' is not available: exec: \"journalctl\": executable file not found in $PATH"
time="2025-07-25T22:58:30+02:00" level=error msg="datasource 'journalctl' is not available: exec: \"journalctl\": executable file not found in $PATH"
time="2025-07-25T22:58:30+02:00" level=error msg="datasource 'journalctl' is not available: exec: \"journalctl\": executable file not found in $PATH"
time="2025-07-25T22:58:30+02:00" level=error msg="datasource 'journalctl' is not available: exec: \"journalctl\": executable file not found in $PATH"
[vic1707@localhost ~]$ podman exec crowdsec cscli collections list
---------------------------------------------------------------------------------------------------------------------
 COLLECTIONS                                                                                                         
---------------------------------------------------------------------------------------------------------------------
 Name                                   📦 Status    Version  Local Path                                             
---------------------------------------------------------------------------------------------------------------------
 crowdsecurity/appsec-generic-rules     ✔️  enabled  1.0      /etc/crowdsec/collections/appsec-generic-rules.yaml    
 crowdsecurity/appsec-virtual-patching  ✔️  enabled  7.4      /etc/crowdsec/collections/appsec-virtual-patching.yaml 
 crowdsecurity/base-http-scenarios      ✔️  enabled  1.2      /etc/crowdsec/collections/base-http-scenarios.yaml     
 crowdsecurity/caddy                    ✔️  enabled  0.1      /etc/crowdsec/collections/caddy.yaml                   
 crowdsecurity/http-cve                 ✔️  enabled  2.9      /etc/crowdsec/collections/http-cve.yaml                
 crowdsecurity/linux                    ✔️  enabled  0.3      /etc/crowdsec/collections/linux.yaml                   
 crowdsecurity/sshd                     ✔️  enabled  0.7      /etc/crowdsec/collections/sshd.yaml                    
 crowdsecurity/whitelist-good-actors    ✔️  enabled  0.2      /etc/crowdsec/collections/whitelist-good-actors.yaml   
---------------------------------------------------------------------------------------------------------------------
[vic1707@localhost ~]$ podman exec crowdsec cscli explain --log "Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27" --type syslog -v
line: Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/cri-logs
        |       ├ 🔴 crowdsecurity/docker-logs
        |       └ 🟢 crowdsecurity/syslog-logs (+12 ~9)
        |               └ update evt.ExpectMode : %!s(int=0) -> 1
        |               └ update evt.Stage :  -> s01-parse
        |               └ update evt.Line.Raw :  -> Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
        |               └ update evt.Line.Src :  -> /tmp/cscli_explain3106532316/cscli_test_tmp.log
        |               └ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2025-07-26 10:35:44.299963582 +0000 UTC
        |               └ create evt.Line.Labels.type : syslog
        |               └ update evt.Line.Process : %!s(bool=false) -> true
        |               └ update evt.Line.Module :  -> file
        |               └ create evt.Parsed.facility : 
        |               └ create evt.Parsed.logsource : syslog
        |               └ create evt.Parsed.message : pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
        |               └ create evt.Parsed.pid : 1029968
        |               └ create evt.Parsed.priority : 
        |               └ create evt.Parsed.program : sshd
        |               └ create evt.Parsed.timestamp : Mar 04 10:06:08
        |               └ create evt.Parsed.timestamp8601 : 
        |               └ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2025-07-26 10:35:44.300632686 +0000 UTC
        |               └ update evt.StrTime :  -> Mar 04 10:06:08
        |               └ create evt.Meta.datasource_path : /tmp/cscli_explain3106532316/cscli_test_tmp.log
        |               └ create evt.Meta.datasource_type : file
        |               └ create evt.Meta.machine : mail.domain.tld
        ├ s01-parse
        |       ├ 🔴 crowdsecurity/appsec-logs
        |       ├ 🔴 crowdsecurity/caddy-logs
        |       └ 🟢 crowdsecurity/sshd-logs (+8 ~1)
        |               └ update evt.Stage : s01-parse -> s02-enrich
        |               └ create evt.Parsed.pam_type : unix
        |               └ create evt.Parsed.sshd_client_ip : 182.59.139.27
        |               └ create evt.Parsed.sshd_invalid_user : 
        |               └ create evt.Parsed.uid : 0
        |               └ create evt.Parsed.euid : 0
        |               └ create evt.Meta.log_type : ssh_failed-auth
        |               └ create evt.Meta.service : ssh
        |               └ create evt.Meta.source_ip : 182.59.139.27
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
        |               ├ create evt.Enriched.MarshaledTime : 2025-03-04T10:06:08Z
        |               ├ update evt.Time : 2025-07-26 10:35:44.300632686 +0000 UTC -> 2025-03-04 10:06:08 +0000 UTC
        |               ├ update evt.MarshaledTime :  -> 2025-03-04T10:06:08Z
        |               ├ create evt.Meta.timestamp : 2025-03-04T10:06:08Z
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |               ├ create evt.Enriched.IsInEU : false
        |               ├ create evt.Enriched.Latitude : 19.074800
        |               ├ create evt.Enriched.Longitude : 72.885600
        |               ├ create evt.Enriched.ASNNumber : 17813
        |               ├ create evt.Enriched.ASNumber : 17813
        |               ├ create evt.Enriched.IsoCode : IN
        |               ├ create evt.Enriched.SourceRange : 182.56.0.0/14
        |               ├ create evt.Enriched.ASNOrg : Mahanagar Telephone Nigam Limited
        |               ├ create evt.Meta.ASNOrg : Mahanagar Telephone Nigam Limited
        |               ├ create evt.Meta.IsoCode : IN
        |               ├ create evt.Meta.ASNNumber : 17813
        |               ├ create evt.Meta.IsInEU : false
        |               ├ create evt.Meta.SourceRange : 182.56.0.0/14
        |       ├ 🔴 crowdsecurity/http-logs
        |       ├ 🟢 crowdsecurity/public-dns-allowlist (unchanged)
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/ssh-bf
                ├ 🟢 crowdsecurity/ssh-bf_user-enum
                ├ 🟢 crowdsecurity/ssh-slow-bf
                └ 🟢 crowdsecurity/ssh-slow-bf_user-enum

The complete, current setup is available in my public git repo if more infos are needed homelab-config/vps/telstar/ignition.bu.yml at moving-out · vic1707/homelab-config · GitHub

Found my answers, there were multiple things wrong with my setup.

  1. journal requires users to use the debian based image <semver>debian
  2. mounted path doesn’t seem to be the right one, /run/log/journal works fine
  3. force_inotify: true can’t be used for journalctl