Crowdsec parser issues

divin31 · January 3, 2025, 11:59am

I’m new to crowdsec and learning the basics, but got stuck.

Running crowdsec in a docker environment on a mac.
My plan is to use a cloudflare and PF bouncer integration. I started with configuring the collections first where I noticed that nothing gets parsed besides non-syslog.

I have started to experiment with a few collections and I’m not sure what I’m doing wrong here or how to troubleshoot further.

docker-compose:

---
services:

 crowdsec:
   image: crowdsecurity/crowdsec:latest
   container_name: crowdsec
   restart: unless-stopped
   environment:
     #this is the list of collections we want to install    
     #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
     COLLECTIONS: "crowdsecurity/http-cve LePresidente/jellyfin firewallservices/pf LePresidente/jellyseerr crowdsecurity/freebsd"
     #GID: "${GID-1000}"
   ports:
    - 8088:8080
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /Applications/docker/crowdsec/crowdsec:/etc/crowdsec
     - /var/log:/var/log 
   networks:
     - crowdsec

networks:
  crowdsec:
    ipam:
      driver: default
      config:
        - subnet: 172.20.0.0/24

/ # cscli collections list

COLLECTIONS
─────────────────────────────────────────────────────────────────────────────────────────
 Name                     📦 Status    Version  Local Path
─────────────────────────────────────────────────────────────────────────────────────────
 crowdsecurity/freebsd    ✔️  enabled  0.3      /etc/crowdsec/collections/freebsd.yaml
 crowdsecurity/http-cve   ✔️  enabled  2.9      /etc/crowdsec/collections/http-cve.yaml
 crowdsecurity/iptables   ✔️  enabled  0.2      /etc/crowdsec/collections/iptables.yaml
 crowdsecurity/linux      ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml
 crowdsecurity/sshd       ✔️  enabled  0.5      /etc/crowdsec/collections/sshd.yaml
 firewallservices/pf      ✔️  enabled  0.2      /etc/crowdsec/collections/pf.yaml
 LePresidente/jellyfin    ✔️  enabled  0.2      /etc/crowdsec/collections/jellyfin.yml
 LePresidente/jellyseerr  ✔️  enabled  0.1      /etc/crowdsec/collections/jellyseerr.yml
─────────────────────────────────────────────────────────────────────────────────────────

/ # cscli collections inspect LePresidente/jellyfin
type: collections
name: LePresidente/jellyfin
file_name: jellyfin.yml
description: 'Jellyfin support : parser and brute-force detection'
author: LePresidente
path: collections/LePresidente/jellyfin.yml
version: "0.2"
parsers:
  - LePresidente/jellyfin-logs
  - crowdsecurity/jellyfin-whitelist
scenarios:
  - LePresidente/jellyfin-bf
local_path: /etc/crowdsec/collections/jellyfin.yml
local_version: "0.2"
local_hash: fe7f6fd1f6dde5ca66020b1d8431784a27dbb9ff34bbd15f4222356eb713a80f
installed: true
downloaded: true
uptodate: true
tainted: false
local: false

Current metrics:

 - (Parser) LePresidente/jellyfin-logs:
╭─────────────────┬──────┬────────┬──────────╮
│ Parsers         │ Hits │ Parsed │ Unparsed │
├─────────────────┼──────┼────────┼──────────┤
│ docker:jellyfin │ 4    │ 0      │ 4        │
╰─────────────────┴──────┴────────┴──────────╯


cscli metrics
Acquisition Metrics:
╭───────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────╮
│ Source            │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├───────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ docker:jellyfin   │ 4          │ -            │ 4              │ -                      │ -                 │
│ docker:jellyseerr │ 24         │ -            │ 24             │ -                      │ -                 │
╰───────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯

Local API Decisions:
╭─────────────────────────────────────────────┬────────┬────────┬───────╮
│ Reason                                      │ Origin │ Action │ Count │
├─────────────────────────────────────────────┼────────┼────────┼───────┤
│ crowdsecurity/CVE-2023-22518                │ CAPI   │ ban    │ 2     │
│ crowdsecurity/ssh-bf                        │ CAPI   │ ban    │ 5422  │
│ crowdsecurity/thinkphp-cve-2018-20062       │ CAPI   │ ban    │ 135   │
│ firewallservices/pf-scan-multi_ports        │ CAPI   │ ban    │ 2310  │
│ crowdsecurity/CVE-2017-9841                 │ CAPI   │ ban    │ 1339  │
│ crowdsecurity/CVE-2023-49103                │ CAPI   │ ban    │ 168   │
│ crowdsecurity/http-cve-2021-42013           │ CAPI   │ ban    │ 8     │
│ crowdsecurity/CVE-2023-22515                │ CAPI   │ ban    │ 39    │
│ crowdsecurity/http-cve-2021-41773           │ CAPI   │ ban    │ 477   │
│ crowdsecurity/ssh-slow-bf                   │ CAPI   │ ban    │ 5510  │
│ crowdsecurity/f5-big-ip-cve-2020-5902       │ CAPI   │ ban    │ 13    │
│ crowdsecurity/vmware-cve-2022-22954         │ CAPI   │ ban    │ 1     │
│ crowdsecurity/vmware-vcenter-vmsa-2021-0027 │ CAPI   │ ban    │ 5     │
│ crowdsecurity/CVE-2019-18935                │ CAPI   │ ban    │ 57    │
│ crowdsecurity/CVE-2024-38475                │ CAPI   │ ban    │ 15    │
│ crowdsecurity/CVE-2024-0012                 │ CAPI   │ ban    │ 1     │
│ crowdsecurity/iptables-scan-multi_ports     │ CAPI   │ ban    │ 1035  │
│ crowdsecurity/fortinet-cve-2018-13379       │ CAPI   │ ban    │ 96    │
│ crowdsecurity/grafana-cve-2021-43798        │ CAPI   │ ban    │ 10    │
│ crowdsecurity/jira_cve-2021-26086           │ CAPI   │ ban    │ 80    │
│ crowdsecurity/CVE-2022-35914                │ CAPI   │ ban    │ 17    │
│ crowdsecurity/apache_log4j2_cve-2021-44228  │ CAPI   │ ban    │ 136   │
│ crowdsecurity/netgear_rce                   │ CAPI   │ ban    │ 98    │
│ crowdsecurity/spring4shell_cve-2022-22965   │ CAPI   │ ban    │ 7     │
│ crowdsecurity/ssh-cve-2024-6387             │ CAPI   │ ban    │ 45    │
│ crowdsecurity/CVE-2022-26134                │ CAPI   │ ban    │ 38    │
│ crowdsecurity/CVE-2022-37042                │ CAPI   │ ban    │ 6     │
╰─────────────────────────────────────────────┴────────┴────────┴───────╯

Local API Metrics:
╭────────────────────┬────────┬──────╮
│ Route              │ Method │ Hits │
├────────────────────┼────────┼──────┤
│ /v1/heartbeat      │ GET    │ 23   │
│ /v1/usage-metrics  │ POST   │ 1    │
│ /v1/watchers/login │ POST   │ 1    │
╰────────────────────┴────────┴──────╯

Local API Machines Metrics:
╭───────────┬───────────────┬────────┬──────╮
│ Machine   │ Route         │ Method │ Hits │
├───────────┼───────────────┼────────┼──────┤
│ localhost │ /v1/heartbeat │ GET    │ 23   │
╰───────────┴───────────────┴────────┴──────╯

Parser Metrics:
╭────────────────────────────────────┬──────┬────────┬──────────╮
│ Parsers                            │ Hits │ Parsed │ Unparsed │
├────────────────────────────────────┼──────┼────────┼──────────┤
│ LePresidente/jellyfin-logs         │ 4    │ -      │ 4        │
│ LePresidente/jellyseerr-logs       │ 24   │ -      │ 24       │
│ child-LePresidente/jellyfin-logs   │ 4    │ -      │ 4        │
│ child-LePresidente/jellyseerr-logs │ 96   │ -      │ 96       │
│ crowdsecurity/non-syslog           │ 28   │ 28     │ -        │
╰────────────────────────────────────┴──────┴────────┴──────────╯

In my acquis.yaml file:

filenames:
  - /var/log/nginx/*.log
  - ./tests/nginx/nginx.log
#this is not a syslog log, indicate which kind of logs it is
labels:
  type: nginx
---
filenames:
 - /var/log/auth.log
 - /var/log/syslog
labels:
  type: syslog
---
filename: /var/log/apache2/*.log
labels:
  type: apache2
---
source: docker
container_name:
 - jellyfin
labels:
  type: jellyfin
---
source: docker
container_name:
 - jellyseerr
labels:
  type: jellyseerr
---
source: docker
container_name:
 - jellyseerr4k
labels:
  type: jellyseerr


line: [22:50:40] [INF] [73] Emby.Server.Implementations.Session.SessionWebSocketListener: Sending ForceKeepAlive message to 1 inactive WebSockets.
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: [23:10:41] [INF] [102] Emby.Server.Implementations.HttpServer.WebSocketManager: WS x.x.x.x closed
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: [00:00:01] [INF] [69] Emby.Server.Implementations.ScheduledTasks.TaskManager: Daily trigger for Playback Reporting Trim Db set to fire at 2025-01-04 00:00:00.000 +00:00, which is 23:59:58.9831331 from now.
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: [02:00:00] [INF] [31] Emby.Server.Implementations.MediaEncoder.EncodingManager: Skipping chapter image extraction for The Undoing as the average chapter duration 0 was lower than the minimum threshold 10000000
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: [21:33:41] [WRN] [72] Emby.Server.Implementations.HttpServer.WebSocketConnection: WS x.x.x.x error receiving data: The remote party closed the WebSocket connection without completing the close handshake.
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: [22:58:16] [INF] [61] Emby.Server.Implementations.Session.SessionWebSocketListener: Sending ForceKeepAlive message to 2 inactive WebSockets.
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: [21:10:34] [INF] [1] Main: Operating system: Debian GNU/Linux 12 (bookworm)
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: [21:10:34] [INF] [1] Emby.Server.Implementations.Plugins.PluginManager: Loaded assembly SQLitePCL.pretty, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null from /config/plugins/Playback Reporting_15.0.0.0/SQLitePCL.pretty.dll
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: [21:10:36] [WRN] [1] Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware: The WebRootPath was not found: /wwwroot. Static files may be unavailable.
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line: System.Net.Http.HttpRequestException: Name or service not known (repo.codyrobibero.dev:443)
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

line:    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
	├ s00-raw
	|	├ 🔴 crowdsecurity/cri-logs
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🔴 crowdsecurity/syslog-logs
	|	└ 🟢 crowdsecurity/non-syslog (+5 ~8)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	├ 🔴 LePresidente/jellyfin-logs
	|	├ 🔴 LePresidente/jellyseerr-logs
	|	├ 🔴 firewallservices/pf-logs
	|	├ 🔴 firewallservices/pf-logs-drop
	|	└ 🔴 crowdsecurity/sshd-logs
	└-------- parser failure 🔴

iiAmLoz · January 3, 2025, 2:01pm

Hey thank you for providing the information.

Please note the collections you are showcasing any parse a specific log line about login failures, this means that all other log lines will be classed as unparsed, so if you attempt to login to the service using a wrong username/password do you see the parsed metric increase?

divin31 · January 3, 2025, 2:16pm

Now I get it. So it only parses failures.
Thank you for the clarification. After I tried some invalid logins, the parsed columns started incrementing.
Sorry for the dumb question, and thanks for the quick answer!

iiAmLoz · January 3, 2025, 2:27pm

Never dumb questions just our documentation does have this section but even for a newbie this is not easy to understand

Topic		Replies	Views
Crowdsec Logs are not parsed	1	436	April 19, 2024
Help with parser failures for caddy logs crowdsec	19	1279	September 22, 2023
Syslog is not getting parsed	3	1462	February 20, 2023
Solution for parsing logs of docker containers crowdsec	5	4463	July 30, 2022
Nextcloud (Docker) - Crowdsec (Docker)	1	95	October 9, 2024

Crowdsec parser issues

Related topics