Help with parser failures for caddy logs

[pwsnla]

Hi,

I installed caddy collection (cscli collections install crowdsecurity/caddy) and it works for the most part but while testing/reviewing results I noticed some parser failures. I’d like to know if there is anything that can be done to eliminate these failures.

I don’t have enough experience with CrowdSec be to understand the reason for these failures an hope that someone here can help. Output of ‘cscli explain --type caddy --verbose’ with sample parser failures is enclosed below (My server IP address was manually replaced). Thank you!

line: {"level":"info","ts":1689610831.4908166,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"53506","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004636,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> {"level":"info","ts":1689610831.4908166,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"53506","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004636,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	|		├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:12.970212353 +0000 UTC
	|		├ create evt.Line.Labels.type : caddy
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : {"level":"info","ts":1689610831.4908166,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"53506","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004636,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	|		├ create evt.Parsed.program : caddy
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:12.971004477 +0000 UTC
	|		├ create evt.Meta.datasource_type : file
	|		├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/caddy-logs
	|	├ 🟢 crowdsecurity/mywhitelists (+1)
	|		├ create evt.Parsed.remote_ip : 91.224.92.16
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: {"level":"info","ts":1689378347.8289995,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"167.94.138.34","remote_port":"40174","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:443","uri":"/","headers":{},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"jellyfin.glosol.com"}},"user_id":"","duration":0.000463121,"size":0,"status":0,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> {"level":"info","ts":1689378347.8289995,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"167.94.138.34","remote_port":"40174","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:443","uri":"/","headers":{},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"jellyfin.glosol.com"}},"user_id":"","duration":0.000463121,"size":0,"status":0,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
	|		├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:12.46327441 +0000 UTC
	|		├ create evt.Line.Labels.type : caddy
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : {"level":"info","ts":1689378347.8289995,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"167.94.138.34","remote_port":"40174","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:443","uri":"/","headers":{},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"jellyfin.glosol.com"}},"user_id":"","duration":0.000463121,"size":0,"status":0,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
	|		├ create evt.Parsed.program : caddy
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:12.463737052 +0000 UTC
	|		├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
	|		├ create evt.Meta.datasource_type : file
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/caddy-logs
	|	├ 🟢 crowdsecurity/mywhitelists (+1)
	|		├ create evt.Parsed.remote_ip : 167.94.138.34
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: {"level":"info","ts":1689540270.8057756,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"167.94.138.124","remote_port":"45920","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.000214521,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> {"level":"info","ts":1689540270.8057756,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"167.94.138.124","remote_port":"45920","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.000214521,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	|		├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:12.811317389 +0000 UTC
	|		├ create evt.Line.Labels.type : caddy
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : {"level":"info","ts":1689540270.8057756,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"167.94.138.124","remote_port":"45920","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.000214521,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	|		├ create evt.Parsed.program : caddy
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:12.811826351 +0000 UTC
	|		├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
	|		├ create evt.Meta.datasource_type : file
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/caddy-logs
	|	├ 🟢 crowdsecurity/mywhitelists (+1)
	|		├ create evt.Parsed.remote_ip : 167.94.138.124
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: {"level":"info","ts":1689807610.132443,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"46.8.29.152","remote_port":"64739","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0","uri":"/","headers":{"Connection":["Keep-Alive"]}},"user_id":"","duration":0.00004416,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> {"level":"info","ts":1689807610.132443,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"46.8.29.152","remote_port":"64739","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0","uri":"/","headers":{"Connection":["Keep-Alive"]}},"user_id":"","duration":0.00004416,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	|		├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:13.523697697 +0000 UTC
	|		├ create evt.Line.Labels.type : caddy
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : {"level":"info","ts":1689807610.132443,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"46.8.29.152","remote_port":"64739","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0","uri":"/","headers":{"Connection":["Keep-Alive"]}},"user_id":"","duration":0.00004416,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	|		├ create evt.Parsed.program : caddy
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:13.524661421 +0000 UTC
	|		├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
	|		├ create evt.Meta.datasource_type : file
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/caddy-logs
	|	├ 🟢 crowdsecurity/mywhitelists (+1)
	|		├ create evt.Parsed.remote_ip : 46.8.29.152
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: {"level":"info","ts":1689484436.395933,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"106.75.71.52","remote_port":"58910","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{"Accept":["*/*"]}},"user_id":"","duration":0.000208601,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> {"level":"info","ts":1689484436.395933,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"106.75.71.52","remote_port":"58910","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{"Accept":["*/*"]}},"user_id":"","duration":0.000208601,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	|		├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:12.745163904 +0000 UTC
	|		├ create evt.Line.Labels.type : caddy
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : {"level":"info","ts":1689484436.395933,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"106.75.71.52","remote_port":"58910","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{"Accept":["*/*"]}},"user_id":"","duration":0.000208601,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	|		├ create evt.Parsed.program : caddy
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:12.745568346 +0000 UTC
	|		├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
	|		├ create evt.Meta.datasource_type : file
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/caddy-logs
	|	├ 🟢 crowdsecurity/mywhitelists (+1)
	|		├ create evt.Parsed.remote_ip : 106.75.71.52
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: {"level":"info","ts":1689628856.60371,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"33370","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004552,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> {"level":"info","ts":1689628856.60371,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"33370","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004552,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	|		├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:13.01135449 +0000 UTC
	|		├ create evt.Line.Labels.type : caddy
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : {"level":"info","ts":1689628856.60371,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"33370","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004552,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"],"Content-Type":[]}}
	|		├ create evt.Parsed.program : caddy
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:13.011642452 +0000 UTC
	|		├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
	|		├ create evt.Meta.datasource_type : file
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/caddy-logs
	|	├ 🟢 crowdsecurity/mywhitelists (+1)
	|		├ create evt.Parsed.remote_ip : 91.224.92.16
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: {"level":"info","ts":1689819255.9251556,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"185.165.190.34","remote_port":"44502","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0","uri":"/.well-known/security.txt","headers":{"Accept-Encoding":["identity"]}},"user_id":"","duration":0.00004096,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/.well-known/security.txt"],"Content-Type":[]}}
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> {"level":"info","ts":1689819255.9251556,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"185.165.190.34","remote_port":"44502","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0","uri":"/.well-known/security.txt","headers":{"Accept-Encoding":["identity"]}},"user_id":"","duration":0.00004096,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/.well-known/security.txt"],"Content-Type":[]}}
	|		├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:13.548887845 +0000 UTC
	|		├ create evt.Line.Labels.type : caddy
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : {"level":"info","ts":1689819255.9251556,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"185.165.190.34","remote_port":"44502","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0","uri":"/.well-known/security.txt","headers":{"Accept-Encoding":["identity"]}},"user_id":"","duration":0.00004096,"size":0,"status":308,"resp_headers":{"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/.well-known/security.txt"],"Content-Type":[]}}
	|		├ create evt.Parsed.program : caddy
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-22 08:35:13.548921485 +0000 UTC
	|		├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
	|		├ create evt.Meta.datasource_type : file
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/caddy-logs
	|	├ 🟢 crowdsecurity/mywhitelists (+1)
	|		├ create evt.Parsed.remote_ip : 185.165.190.34
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

[iiAmLoz]

Are you on the latest version of CrowdSec?

These IP’s are classed as failing to parse because they are in your whitelist, not because the line fails. However, the caddy parser is due an update to 1.5 standards as we have created a json parser module which is faster and can avoid these issue of edge cases.

I will work on this within the following days. If these lines are infact different IP’s and not all yours from your whitelist let me know.

[pwsnla]

These IPs are NOT in my whitelist. I can remove the whitelist and try the parse again just to see if the whitelist is indeed somehow causing a problem.

Here is the output of cscli show version on my system. I regularly update my system but would get whatever is bundled with ubuntu updates, which may not be latest and greatest:

2023/07/25 05:25:30 version: v1.5.2-debian-pragmatic-4fbc3402fba932c8bd34b671527dcf7909d264c0
2023/07/25 05:25:30 Codename: alphaga
2023/07/25 05:25:30 BuildDate: 2023-05-26_16:14:02
2023/07/25 05:25:30 GoVersion: 1.20.1
2023/07/25 05:25:30 Platform: linux
2023/07/25 05:25:30 Constraint_parser: >= 1.0, <= 2.0
2023/07/25 05:25:30 Constraint_scenario: >= 1.0, < 3.0
2023/07/25 05:25:30 Constraint_api: v1
2023/07/25 05:25:30 Constraint_acquis: >= 1.0, < 2.0

Thank you for your help!

[pwsnla]

Just to eliminate it as a possible source of parser failures I removed my whitelist and the parser still fails occasionally, Here is a sample failure without whitelist:

line: {"level":"info","ts":1690267717.8317127,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"175.107.1.113","remote_port":"52184","proto":"HTTP/1.0","method":"GET","host":"","uri":"/boaform/admin/formLogin?us>
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (+5 ~8)
        |               ├ update evt.ExpectMode : %!s(int=0) -> 1
        |               ├ update evt.Stage :  -> s01-parse
        |               ├ update evt.Line.Raw :  -> {"level":"info","ts":1690267717.8317127,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"175.107.1.113","remote_port":"52184","proto":"HTTP/1.0","method":"GET>
        |               ├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
        |               ├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-25 07:04:35.420130487 +0000 UTC
        |               ├ create evt.Line.Labels.type : caddy
        |               ├ update evt.Line.Process : %!s(bool=false) -> true
        |               ├ update evt.Line.Module :  -> file
        |               ├ create evt.Parsed.program : caddy
        |               ├ create evt.Parsed.message : {"level":"info","ts":1690267717.8317127,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"175.107.1.113","remote_port":"52184","proto":"HTTP/1.0","method":"G>
        |               ├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-25 07:04:35.420863051 +0000 UTC
        |               ├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
        |               ├ create evt.Meta.datasource_type : file
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       ├ 🔴 crowdsecurity/caddy-logs
        |       └ 🔴 crowdsecurity/sshd-logs
        └-------- parser failure 🔴

Common pattern for these failures seems to be the “crowdsecurity/sshd-logs” step. I am not sure why sshd-logs are involved here.

For comparison here is a successful parse example:

line: {"level":"info","ts":1690268478.6025789,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"192.241.225.57","remote_port":"34798","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0","uri":"/","headers":{"User>
        ├ s00-raw
        |       ├ 🟢 crowdsecurity/non-syslog (+5 ~8)
        |               ├ update evt.ExpectMode : %!s(int=0) -> 1
        |               ├ update evt.Stage :  -> s01-parse
        |               ├ update evt.Line.Raw :  -> {"level":"info","ts":1690268478.6025789,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"192.241.225.57","remote_port":"34798","proto":"HTTP/1.1","method":"GE>
        |               ├ update evt.Line.Src :  -> /mnt/data/work/caddy/log/access.log
        |               ├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-25 07:04:35.421482814 +0000 UTC
        |               ├ create evt.Line.Labels.type : caddy
        |               ├ update evt.Line.Process : %!s(bool=false) -> true
        |               ├ update evt.Line.Module :  -> file
        |               ├ create evt.Parsed.message : {"level":"info","ts":1690268478.6025789,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"192.241.225.57","remote_port":"34798","proto":"HTTP/1.1","method":">
        |               ├ create evt.Parsed.program : caddy
        |               ├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-07-25 07:04:35.421789975 +0000 UTC
        |               ├ create evt.Meta.datasource_path : /mnt/data/work/caddy/log/access.log
        |               ├ create evt.Meta.datasource_type : file
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/caddy-logs (+12 ~1)
        |               └ update evt.Stage : s01-parse -> s02-enrich
        |               └ create evt.Parsed.remote_ip : 192.241.225.57
        |               └ create evt.Parsed.request : /
        |               └ create evt.Parsed.verb : GET
        |               └ create evt.Parsed.http_user_agent : Mozilla/5.0 zgrab/0.x
        |               └ create evt.Meta.http_user_agent : Mozilla/5.0 zgrab/0.x
        |               └ create evt.Meta.service : http
        |               └ create evt.Meta.log_type : http_access-log
        |               └ create evt.Meta.source_ip : 192.241.225.57
        |               └ create evt.Meta.target_fqdn : 0.0.0.0
        |               └ create evt.Meta.http_path : /
        |               └ create evt.Meta.http_status : 308
        |               └ create evt.Meta.http_verb : GET
        ├ s02-enrich
        |       ├ 🔴 crowdsecurity/dateparse-enrich
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |               ├ create evt.Enriched.Longitude : -122.383800
        |               ├ create evt.Enriched.SourceRange : 192.241.128.0/17
        |               ├ create evt.Enriched.ASNNumber : 14061
        |               ├ create evt.Enriched.ASNOrg : DIGITALOCEAN-ASN
        |               ├ create evt.Enriched.ASNumber : 14061
        |               ├ create evt.Enriched.IsInEU : false
        |               ├ create evt.Enriched.IsoCode : US
        |               ├ create evt.Enriched.Latitude : 37.730800
        |               ├ create evt.Meta.IsInEU : false
        |               ├ create evt.Meta.SourceRange : 192.241.128.0/17
        |               ├ create evt.Meta.ASNOrg : DIGITALOCEAN-ASN
        |               ├ create evt.Meta.ASNNumber : 14061
        |               ├ create evt.Meta.IsoCode : US
        |       ├ 🟢 crowdsecurity/http-logs (+6)
        |               ├ create evt.Parsed.file_ext :
        |               ├ create evt.Parsed.file_frag :
        |               ├ create evt.Parsed.impact_completion : true
        |               ├ create evt.Parsed.static_ressource : false
        |               ├ create evt.Parsed.file_dir : /
        |               ├ create evt.Meta.http_args_len : 0
        |       └ 🟢 crowdsecurity/whitelists (unchanged)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/http-bad-user-agent
                └ 🟢 crowdsecurity/http-crawl-non_statics

I hope this helps.

[iiAmLoz]

Common pattern for these failures seems to be the “crowdsecurity/sshd-logs” step. I am not sure why sshd-logs are involved here.

They are not involved the way cscli explains shows the data is it attempts to do the next parser if the one before fails and because caddy failed it then went to sshd which would automatically fail because it doesnt pass the filter.

I have started working on a update for caddy parser like I said I want to have out shortly.

[pwsnla]

OK. I wasn’t sure how to read the output from explain. Thank you for the explanation.

[pwsnla]

Hi @iiAmLoz,

I was wondering if there is any progress on the caddy parser update you mentioned

[iiAmLoz]

Yes I apologise, I got side tracked. There few things we are debugging but hopefully should be out end of month.

[iiAmLoz]

Caddy update 1.5

[iiAmLoz]

Using your top example you can now see it passes the parser

$ cscli explain --log '{"level":"info","ts":1689610831.4908166,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"53506","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004636,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}' --type caddy -v
line: {"level":"info","ts":1689610831.4908166,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"53506","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004636,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	├ s00-raw
	|	├ 🟢 crowdsecurity/non-syslog (+5 ~8)
	|		├ update evt.ExpectMode : %!s(int=0) -> 1
	|		├ update evt.Stage :  -> s01-parse
	|		├ update evt.Line.Raw :  -> {"level":"info","ts":1689610831.4908166,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"53506","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004636,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	|		├ update evt.Line.Src :  -> /tmp/cscli_explain3770943202/cscli_test_tmp.log
	|		├ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-08-17 10:57:47.958796759 +0000 UTC
	|		├ create evt.Line.Labels.type : caddy
	|		├ update evt.Line.Process : %!s(bool=false) -> true
	|		├ update evt.Line.Module :  -> file
	|		├ create evt.Parsed.message : {"level":"info","ts":1689610831.4908166,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"91.224.92.16","remote_port":"53506","proto":"HTTP/1.1","method":"GET","host":"0.0.0.0:80","uri":"/","headers":{}},"user_id":"","duration":0.00004636,"size":0,"status":308,"resp_headers":{"Content-Type":[],"Server":["Caddy"],"Connection":["close"],"Location":["https://0.0.0.0/"]}}
	|		├ create evt.Parsed.program : caddy
	|		├ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-08-17 10:57:47.958863395 +0000 UTC
	|		├ create evt.Meta.datasource_path : /tmp/cscli_explain3770943202/cscli_test_tmp.log
	|		├ create evt.Meta.datasource_type : file
	├ s01-parse
	|	├ 🟢 crowdsecurity/caddy-logs (+11 ~2)
	|		├ update evt.Stage : s01-parse -> s02-enrich
	|		├ create evt.Parsed.remote_ip : 91.224.92.16
	|		├ create evt.Parsed.request : /
	|		├ create evt.Parsed.verb : GET
	|		├ create evt.Unmarshaled.caddy : map[duration:4.636e-05 level:info logger:http.log.access msg:handled request request:map[headers:map[] host:0.0.0.0:80 method:GET proto:HTTP/1.1 remote_ip:91.224.92.16 remote_port:53506 uri:/] resp_headers:map[Connection:[close] Content-Type:[] Location:[https://0.0.0.0/] Server:[Caddy]] size:0 status:308 ts:1.6896108314908166e+09 user_id:]
	|		├ update evt.StrTime :  -> 1689610831
	|		├ create evt.Meta.http_path : /
	|		├ create evt.Meta.http_status : 308
	|		├ create evt.Meta.service : http
	|		├ create evt.Meta.source_ip : 91.224.92.16
	|		├ create evt.Meta.target_fqdn : 0.0.0.0:80
	|		├ create evt.Meta.http_verb : GET
	|		├ create evt.Meta.log_type : http_access-log
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+2 ~2)
	|		├ create evt.Enriched.MarshaledTime : 2023-07-17T17:20:31+01:00
	|		├ update evt.Time : 2023-08-17 10:57:47.958863395 +0000 UTC -> 2023-07-17 17:20:31 +0100 BST
	|		├ update evt.MarshaledTime :  -> 2023-07-17T17:20:31+01:00
	|		├ create evt.Meta.timestamp : 2023-07-17T17:20:31+01:00
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|		├ create evt.Enriched.ASNNumber : 209605
	|		├ create evt.Enriched.Latitude : 51.496400
	|		├ create evt.Enriched.SourceRange : 91.224.92.0/24
	|		├ create evt.Enriched.Longitude : -0.122400
	|		├ create evt.Enriched.ASNOrg : UAB Host Baltic
	|		├ create evt.Enriched.ASNumber : 209605
	|		├ create evt.Enriched.IsInEU : false
	|		├ create evt.Enriched.IsoCode : GB
	|		├ create evt.Meta.IsInEU : false
	|		├ create evt.Meta.IsoCode : GB
	|		├ create evt.Meta.SourceRange : 91.224.92.0/24
	|		├ create evt.Meta.ASNNumber : 209605
	|		├ create evt.Meta.ASNOrg : UAB Host Baltic
	|	├ 🟢 crowdsecurity/http-logs (+6)
	|		├ create evt.Parsed.impact_completion : true
	|		├ create evt.Parsed.static_ressource : false
	|		├ create evt.Parsed.file_dir : /
	|		├ create evt.Parsed.file_ext : 
	|		├ create evt.Parsed.file_frag : 
	|		├ create evt.Meta.http_args_len : 0
	|	├ 🟢 crowdsecurity/jellyfin-whitelist (unchanged)
	|	├ 🟢 crowdsecurity/nextcloud-whitelist (unchanged)
	|	└ 🟢 crowdsecurity/whitelists (unchanged)
	├-------- parser success 🟢
	├ Scenarios
		└ 🟢 crowdsecurity/http-crawl-non_statics

[pwsnla]

Thank you @iiAmLoz !

I look forward to trying this out. Is this new version available to install already?

[iiAmLoz]

Nope it seems I found a bug untested feature in hubtest. Cause it converts the compared value to a string it does not match some values, so this needs to be patched.

[pwsnla]

Thanks for the update. I’ll wait until the problem you found is patched.

[pwsnla]

Hi @iiAmLoz,

I found what I think might be another potential problem with parsing caddy logs. This scenario has to do with using a Cloudflare proxy, which is pretty common since a lot of people use Cloudflare for DNS management and using Cloudflare proxy offers some benefits. Caddy recent version introduced a lot of features to handle such proxy scenarios and still be able to get and log real client IP. This is mentioned here:

caddyhttp: Determine real client IP if trusted proxies configured

Caddy can now be configured to prevent spoofing by using Cloudflare’s CF-Connecting-IP to populate client_ip. With this implemented caddy logs Cloudflare proxy IP address as remote_ip and the real client IP is logged as client_ip.

Doing some preliminary testing it appears that crowdsec caddy log parser (at least in the current version) looks at remote_ip and not client_ip, which may lead to Cloudflare IPs being banned, which is not what should happen. Client_ip is always populated and outside of “trusted proxy” scenario client_ip will be the same as remote_ip.

I thought I would mention this since you are looking at Caddy parser now.

[iiAmLoz]

Thank you, last time I checked the PR was not merged so I will look at the parser now.

Any chance you can provide some log samples? as our tests dont have client_ip it could be old logs.

[pwsnla]

This new caddy feature has been released now. Yes, I can provide a sample of logs with client_ip. I do have a site with Cloudflare proxy set up now.

What’s the best way to get this log to you? The log is not huge but I don’t see a way to attach a text file here.

[iiAmLoz]

You can email me laurence@crowdsec.net

[pwsnla]

@iiAmLoz

laurence@crowdsec.net

Done! Let me know if you need anything else.

[pwsnla]

Hi @iiAmLoz,

I wanted to check to see if there is any progress with updates for Caddy parser.
Thank you!

[iiAmLoz]

Hey

So the current update Caddy update 1.5 by LaurenceJJones · Pull Request #806 · crowdsecurity/hub · GitHub works but there a bug in our CI pipeline NOT a bug with using it within crowdsec. Within the PR you see a link a CrowdSec issue which outlines it more.

So im stuck as I cant merge it and it will continue to break our tests (it will work within your / everyones crowdsec) but we cant knowingly break the test pipeline. We have it planned to form a plan around this but we had 1.5.3/4 to deal with firstly