Agent fails to parse logs - but cscli explain works

crowdsec
1

On a kubernetes cluster, everything looks working fine except agents fails parsing nginx logs.

Running cscli explain inside the agent container shows no errors.

What can it be?

Agent logs:

time="24-01-2023 18:37:16" level=debug msg="pushing {Raw:2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] \"GET /api/live/ws HTTP/1.1\" 400 12 \"-\" \"Mozilla/5.0 (X11; Linux
x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36\" \"-\" Src:/var/log/containers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34c
bf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log Time:2023-01-24 18:37:16.203161076 +0000 UTC m=+2349.726670066 Labels:map[program:nginx type:docker] Process:true Module:file}" tail=/var/log/cont
ainers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34cbf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log type=file
time="24-01-2023 18:37:16" level=debug msg="+ Processing 3 statics" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="[time] doesn't exist"
time="24-01-2023 18:37:16" level=debug msg="Empty value for evt.StrTime, skip." id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="[log] doesn't exist"
time="24-01-2023 18:37:16" level=debug msg="Empty value for .Parsed[message], skip." id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg=".Parsed[program] = 'nginx'" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ok" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="move Event from stage s00-raw to s01-parse" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="node successful, stop end stage s00-raw" node-name=black-leaf stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="(black-snow) target field 'message' doesn't exist in map[program:nginx]" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="+ Grok '(%{IP...' didn't return data on ''" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="(wandering-snow) target field 'message' doesn't exist in map[program:nginx]" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="+ Grok '(%{IP...' didn't return data on ''" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=falling-thunder name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=late-moon name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=divine-shadow name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=little-water name=crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=hidden-sky name=crowdsecurity/sshd-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Log didn't finish stage s01-parse"
time="24-01-2023 18:37:16" level=debug msg="Discarding line {Type:0 ExpectMode:0 Whitelisted:false WhitelistReason: Stage:s01-parse Line:{Raw:2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] \"GET /api/live/ws HTTP/1.1\" 400 12 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36\" \"-\" Src:/var/log/containers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34cbf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log Time:2023-01-24 18:37:16.203161076 +0000 UTC m=+2349.726670066 Labels:map[program:nginx type:docker] Process:true Module:file} Parsed:map[program:nginx] Enriched:map[] Overflow:{Mapkey: BucketId: Whitelisted:false Reprocess:false Sources:map[] Alert:<nil> APIAlerts:[]} Time:2023-01-24 18:37:16.203255925 +0000 UTC StrTime: StrTimeFormat: MarshaledTime: Process:false Meta:map[]}"

cscli explain

cscli explain --log '2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] "GET /api/live/ws HTTP/1.1" 400 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"' --debug --type nginx
DEBU[24-01-2023 18:38:21] Using /etc/crowdsec/config.yaml as configuration file
line: 2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] "GET /api/live/ws HTTP/1.1" 400 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/docker-logs
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/nginx-logs (+23 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |       └ 🟢 crowdsecurity/http-logs (+7)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/http-crawl-non_statics
                └ 🟢 crowdsecurity/http-probing

Content of /etc/crowdsec/acquis.yaml

filenames:
  - /var/log/containers/*-nginx-ingress-*_*_*.log
force_inotify: true
labels:
  type: docker
  program: nginx
2

The problem is related with Crowdsec agent cannot parse Nginx ingress logs unless "container_runtime: nginx" · Issue #66 · crowdsecurity/helm-charts · GitHub

3

For future travelers, if you are running on kubernetes and your runtime is not docker, you need to setup the correct type here. In my case was containerd.

If you are using helm, make sure to override this variable on your values.yaml.