Journalctl parsers fails (solved)

[bjo]

Hi,

crowdsec.log tells me:

time="04-03-2023 00:00:09" level=info msg="Running journalctl command: /usr/bin/journalctl [journalctl --follow -n 0 _SYSTEMD_UNIT=dovecot.service]" src="journalctl-_SYSTEMD_UNIT=dovecot.service" type=journalctl
time="04-03-2023 00:00:09" level=info msg="Running journalctl command: /usr/bin/journalctl [journalctl --follow -n 0 _SYSTEMD_UNIT=sshd.service]" src="journalctl-_SYSTEMD_UNIT=sshd.service" type=journalctl
time="04-03-2023 00:00:09" level=info msg="Running journalctl command: /usr/bin/journalctl [journalctl --follow -n 0 _SYSTEMD_UNIT=postfix.service]" src="journalctl-_SYSTEMD_UNIT=postfix.service" type=journalctl

but lines like

Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): check pass; user unknown
Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_faillock(sshd:auth): User unknown
Mar 04 10:06:10 mail.domain.tld sshd[1029968]: Failed password for invalid user clare from 182.59.139.27 port 34293 ssh2
Mar 04 10:06:11 mail.domain.tld sshd[1029968]: Received disconnect from 182.59.139.27 port 34293:11: Bye Bye [preauth]
Mar 04 10:06:11 mail.domain.tld sshd[1029968]: Disconnected from invalid user clare 182.59.139.27 port 34293 [preauth]
Mar 04 10:09:45 mail.domain.tld sshd[1030304]: Invalid user account from 182.59.139.27 port 60228

get ignored.

Explaining also fails - regardless if type is syslog or journalctl

line: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
	├ s01-parse
	|	├ 🔴 crowdsecurity/dovecot-logs
	|	├ 🔴 crowdsecurity/nextcloud-logs
	|	├ 🔴 crowdsecurity/nginx-logs
	|	├ 🔴 crowdsecurity/postfix-logs
	|	├ 🔴 crowdsecurity/postscreen-logs
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

[iiAmLoz]

Type syslog needs the “whole syslog line” not just the message part as it needs to parse sshd from the line example:

╰─λ sudo cscli explain --log "Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27" --type syslog -v --only-successful-parsers
line: Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
├ s00-raw
|       └ 🟢 crowdsecurity/syslog-logs (+12 ~9)
|               └ update evt.ExpectMode : %!s(int=0) -> 1
|               └ update evt.Stage :  -> s01-parse
|               └ update evt.Line.Raw :  -> Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
|               └ update evt.Line.Src :  -> /tmp/cscli_test_tmp.log
|               └ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-03-05 20:14:27.808427683 +0000 UTC
|               └ create evt.Line.Labels.type : syslog
|               └ update evt.Line.Process : %!s(bool=false) -> true
|               └ update evt.Line.Module :  -> file
|               └ create evt.Parsed.facility :
|               └ create evt.Parsed.logsource : syslog
|               └ create evt.Parsed.message : pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
|               └ create evt.Parsed.pid : 1029968
|               └ create evt.Parsed.priority :
|               └ create evt.Parsed.program : sshd
|               └ create evt.Parsed.timestamp : Mar 04 10:06:08
|               └ create evt.Parsed.timestamp8601 :
|               └ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-03-05 20:14:27.808733522 +0000 UTC
|               └ update evt.StrTime :  -> Mar 04 10:06:08
|               └ create evt.Meta.datasource_path : /tmp/cscli_test_tmp.log
|               └ create evt.Meta.datasource_type : file
|               └ create evt.Meta.machine : mail.domain.tld
├ s01-parse
|       ├ 🟢 crowdsecurity/sshd-logs (+8 ~1)
|               ├ update evt.Stage : s01-parse -> s02-enrich
|               ├ create evt.Parsed.pam_type : unix
|               ├ create evt.Parsed.sshd_invalid_user :
|               ├ create evt.Parsed.euid : 0
|               ├ create evt.Parsed.sshd_client_ip : 182.59.139.27
|               ├ create evt.Parsed.uid : 0
|               ├ create evt.Meta.service : ssh
|               ├ create evt.Meta.source_ip : 182.59.139.27
|               ├ create evt.Meta.log_type : ssh_failed-auth
├ s02-enrich
|       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
|               ├ create evt.Enriched.MarshaledTime : 2023-03-04T10:06:08Z
|               ├ update evt.MarshaledTime :  -> 2023-03-04T10:06:08Z
|               ├ create evt.Meta.timestamp : 2023-03-04T10:06:08Z
|       ├ 🟢 crowdsecurity/geoip-enrich (+13)
|               ├ create evt.Enriched.SourceRange : 182.56.0.0/14
|               ├ create evt.Enriched.ASNOrg : Mahanagar Telephone Nigam Limited
|               ├ create evt.Enriched.IsoCode : IN
|               ├ create evt.Enriched.Latitude : 19.074800
|               ├ create evt.Enriched.ASNNumber : 17813
|               ├ create evt.Enriched.ASNumber : 17813
|               ├ create evt.Enriched.IsInEU : false
|               ├ create evt.Enriched.Longitude : 72.885600
|               ├ create evt.Meta.ASNNumber : 17813
|               ├ create evt.Meta.ASNOrg : Mahanagar Telephone Nigam Limited
|               ├ create evt.Meta.SourceRange : 182.56.0.0/14
|               ├ create evt.Meta.IsInEU : false
|               ├ create evt.Meta.IsoCode : IN
|       └ 🟢 crowdsecurity/whitelists (unchanged)
├-------- parser success 🟢├ Scenarios
├ 🟢 crowdsecurity/ssh-bf
├ 🟢 crowdsecurity/ssh-bf_user-enum
├ 🟢 crowdsecurity/ssh-slow-bf
└ 🟢 crowdsecurity/ssh-slow-bf_user-enum

Using cscli metrics can provide information on how much is parsed

[bjo]

The whole line unfortunately also fails:

cscli explain --log "Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27" --type syslog -v                         
line: Mar 04 10:06:08 mail.domain.tld sshd[1029968]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=182.59.139.27
	├ s01-parse
	|	├ 🔴 crowdsecurity/dovecot-logs
	|	├ 🔴 crowdsecurity/nextcloud-logs
	|	├ 🔴 crowdsecurity/nginx-logs
	|	├ 🔴 crowdsecurity/postfix-logs
	|	├ 🔴 crowdsecurity/postscreen-logs
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

[iiAmLoz]

Seems you dont have s00 parser? can you run cscli collections install crowdsecurity/linux

[bjo]

Yes, seems this was the issue. Thanks, it works now!