Help: caddy-logs parser failure

crowdsec
1

As per title I’m trying to setup the caddy-logs / caddy collection on my instance but lookig at metrics and doing cscli explain i keep getting 0 lines parsed and parser failure, i’ll post my config to see if im doing things wrong (which i probably am) or if it’s a bug

cscli config show

Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
cscli:
  - Output                  : human
  - Hub Branch              :
  - Hub Folder              : /etc/crowdsec/hub
Local API Server:
  - Listen URL              : 127.0.0.1:8080
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

cscli metrics

INFO[25-12-2021 09:52:04 PM] Acquisition Metrics:
+------------------------------------------+------------+--------------+----------------+------------------------+
|                  SOURCE                  | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+------------------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/auth.log                   |         82 | -            |             82 | -                      |
| file:/var/log/caddy/access-bitwarden.log |         51 | -            |             51 | -                      |
| file:/var/log/caddy/access-dns.log       |       1106 | -            |           1106 | -                      |
+------------------------------------------+------------+--------------+----------------+------------------------+
INFO[25-12-2021 09:52:04 PM] Local Api Metrics:
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    |    5 |
| /v1/alerts/1         | GET    |    1 |
| /v1/decisions/stream | GET    |  806 |
| /v1/watchers/login   | POST   |   18 |
+----------------------+--------+------+
INFO[25-12-2021 09:52:04 PM] Local Api Machines Metrics:
+----------------------------------+--------------+--------+------+
|             MACHINE              |    ROUTE     | METHOD | HITS |
+----------------------------------+--------------+--------+------+
| 4a762d65fa674af2840afb7233aa1929 | /v1/alerts/1 | GET    |    1 |
| 4a762d65fa674af2840afb7233aa1929 | /v1/alerts   | GET    |    5 |
+----------------------------------+--------------+--------+------+
INFO[25-12-2021 09:52:04 PM] Local Api Bouncers Metrics:
+------------------------------+----------------------+--------+------+
|           BOUNCER            |        ROUTE         | METHOD | HITS |
+------------------------------+----------------------+--------+------+
| cs-firewall-bouncer-2hR7zgxz | /v1/decisions/stream | GET    |  806 |
+------------------------------+----------------------+--------+------+

cscli parsers list

-------------------------------------------------------------------------------------------------
 NAME                      📦 STATUS   VERSION  LOCAL PATH
-------------------------------------------------------------------------------------------------
 crowdsecurity/caddy-logs  ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/caddy-logs.yaml
 crowdsecurity/http-logs   ✔️  enabled  0.6      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
 crowdsecurity/whitelists  ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml
 crowdsecurity/sshd-logs   ✔️  enabled  1.6      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
-------------------------------------------------------------------------------------------------

cscli scenarios list

----------------------------------------------------------------------------------------------------------------------------
 NAME                                        📦 STATUS   VERSION  LOCAL PATH
----------------------------------------------------------------------------------------------------------------------------
 crowdsecurity/http-backdoors-attempts       ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml
 crowdsecurity/iptables-scan-multi_ports     ✔️  enabled  0.1      /etc/crowdsec/scenarios/iptables-scan-multi_ports.yaml
 crowdsecurity/http-bad-user-agent           ✔️  enabled  0.4      /etc/crowdsec/scenarios/http-bad-user-agent.yaml
 crowdsecurity/http-open-proxy               ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-open-proxy.yaml
 crowdsecurity/ssh-slow-bf                   ✔️  enabled  0.2      /etc/crowdsec/scenarios/ssh-slow-bf.yaml
 crowdsecurity/http-generic-bf               ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-generic-bf.yaml
 crowdsecurity/http-probing                  ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-probing.yaml
 crowdsecurity/http-sensitive-files          ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml
 crowdsecurity/http-xss-probing              ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml
 crowdsecurity/http-path-traversal-probing   ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml
 crowdsecurity/apache_log4j2_cve-2021-44228  ✔️  enabled  0.4      /etc/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml
 crowdsecurity/http-sqli-probing             ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml
 ltsich/http-w00tw00t                        ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-w00tw00t.yaml
 crowdsecurity/ssh-bf                        ✔️  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml
 crowdsecurity/http-crawl-non_statics        ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml
----------------------------------------------------------------------------------------------------------------------------

cscli collections list

------------------------------------------------------------------------------------------------------------
 NAME                               📦 STATUS   VERSION  LOCAL PATH
------------------------------------------------------------------------------------------------------------
 crowdsecurity/base-http-scenarios  ✔️  enabled  0.5      /etc/crowdsec/collections/base-http-scenarios.yaml
 crowdsecurity/caddy                ✔️  enabled  0.1      /etc/crowdsec/collections/caddy.yaml
 crowdsecurity/sshd                 ✔️  enabled  0.2      /etc/crowdsec/collections/sshd.yaml
------------------------------------------------------------------------------------------------------------

cscli explain --file /var/log/caddy/access-bitwarden.log --type caddy (i’ve removed the IP’s and domains manually)

line: {"level":"info","ts":1640457680.3688867,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"0.0.0.0:43482","proto":"HTTP/2.0","method":"GET","host":"host.uri.here","uri":"/icons/app.crowdsec.net/icon.png","headers":{"Accept-Encoding":["gzip"],"Cf-Ipcountry":["IT"],"Cf-Ray":["6c3433f31b9f71f0-LHR"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"96\", \"Google Chrome\";v=\"96\""],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["0.0.0.0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36"],"Sec-Ch-Ua-Platform":["\"Windows\""],"Sec-Fetch-Site":["none"],"X-Forwarded-Proto":["https"],"Cdn-Loop":["cloudflare"],"Cf-Connecting-Ip":["0.0.0.0"],"Dnt":["1"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Accept-Language":["en,it;q=0.9"],"Cookie":["__cf_bm=6d.0AIqDzRSepqb5r70l5stpAaEloXuabF5ONbuVbgk-1640456861-0-AQcvUKaqaCBH/NUthwyLb/Nu20vt5aJtk6xY3Ng05mfdV3HtF9K+/NYzueiwuZo2GXVPLUnz4lmifoDraKyMa66kG33+i+7Q8CGp7QcBVSFwjZ3VDdk+g072E58Aj+fmKw=="]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"host.uri.here"}},"common_log":"0.0.0.0 - - [25/Dec/2021:18:41:20 +0000] \"GET /icons/app.crowdsec.net/icon.png HTTP/2.0\" 200 64313","user_id":"","duration":0.405881422,"size":64313,"status":200,"resp_headers":{"Feature-Policy":["accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'none'; camera 'none'; encrypted-media 'none'; fullscreen 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture 'none'; sync-xhr 'self' https://haveibeenpwned.com https://2fa.directory; usb 'none'; vr 'none'"],"Referrer-Policy":["same-origin"],"X-Frame-Options":["SAMEORIGIN"],"X-Xss-Protection":["1; mode=block"],"X-Content-Type-Options":["nosniff"],"Content-Security-Policy":["frame-ancestors 'self' chrome-extension://nngceckbapebfimnlniiiahkandclblb chrome-extension://jbkfoedolllekgbhcbcoahefnbanhhlh moz-extension://* ;"],"Content-Length":["64313"],"Date":["Sat, 25 Dec 2021 18:41:20 GMT"],"Content-Type":["image/png"],"Server":["Caddy","Rocket"],"Cache-Control":["public, immutable, max-age=2592000"]}}
        ├ s01-parse
        |       ├ 🔴 crowdsecurity/caddy-logs
        |       └ 🔴 crowdsecurity/sshd-logs
        └-------- parser failure 🔴
2

Hello, it is weird that the crowdsecurity/linux collection is not installed. This one should be installed automatically when installing crowdsec.

Can you please install this collection by running sudo cscli hub update and sudo cscli collections install crowdsecurity/linux , reload the crowdsec service and retry please?

3

Well would you look at that, it works now! :crazy_face:

This one should be installed automatically when installing crowdsec.

It didn’t because i’ve compiled manually from source since i’m running on a Raspberry Pi 4
Thanks for the help :muscle:

4

Ok cool :slight_smile:

Just for your information, the parser crowdsecurity/syslog-logs is mandatory to get your log parsed.

5

and as well, we’ll distribute packages for raspbian in the next release (along with debian armhf) :slight_smile:

6

This same situation happened to me in 2024. I’m using crowdsec docker container. I passed in collections via environment variables. I didn’t have crowdsecurity/linux. Also was trying to figure out why caddy logs were not parsing. Glad I found this thread. Thanks! :slight_smile: