On a Kubernetes cluster, I installed crowdsec and a bouncer for nginx ingress.
Everything appears to work, except logs are not parsed. However, inside an agent container, when running cscli explain for the log line, it works. What can it be?
agent logs:
time="24-01-2023 18:37:16" level=debug msg="pushing {Raw:2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] \"GET /api/live/ws HTTP/1.1\" 400 12 \"-\" \"Mozilla/5.0 (X11; Linux
x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36\" \"-\" Src:/var/log/containers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34c
bf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log Time:2023-01-24 18:37:16.203161076 +0000 UTC m=+2349.726670066 Labels:map[program:nginx type:docker] Process:true Module:file}" tail=/var/log/cont
ainers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34cbf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log type=file
time="24-01-2023 18:37:16" level=debug msg="+ Processing 3 statics" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="[time] doesn't exist"
time="24-01-2023 18:37:16" level=debug msg="Empty value for evt.StrTime, skip." id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="[log] doesn't exist"
time="24-01-2023 18:37:16" level=debug msg="Empty value for .Parsed[message], skip." id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg=".Parsed[program] = 'nginx'" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ok" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="move Event from stage s00-raw to s01-parse" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="node successful, stop end stage s00-raw" node-name=black-leaf stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="(black-snow) target field 'message' doesn't exist in map[program:nginx]" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="+ Grok '(%{IP...' didn't return data on ''" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="(wandering-snow) target field 'message' doesn't exist in map[program:nginx]" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="+ Grok '(%{IP...' didn't return data on ''" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=falling-thunder name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=late-moon name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=divine-shadow name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=little-water name=crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=hidden-sky name=crowdsecurity/sshd-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Log didn't finish stage s01-parse"
time="24-01-2023 18:37:16" level=debug msg="Discarding line {Type:0 ExpectMode:0 Whitelisted:false WhitelistReason: Stage:s01-parse Line:{Raw:2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] \"GET /api/live/ws HTTP/1.1\" 400 12 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36\" \"-\" Src:/var/log/containers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34cbf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log Time:2023-01-24 18:37:16.203161076 +0000 UTC m=+2349.726670066 Labels:map[program:nginx type:docker] Process:true Module:file} Parsed:map[program:nginx] Enriched:map[] Overflow:{Mapkey: BucketId: Whitelisted:false Reprocess:false Sources:map[] Alert:<nil> APIAlerts:[]} Time:2023-01-24 18:37:16.203255925 +0000 UTC StrTime: StrTimeFormat: MarshaledTime: Process:false Meta:map[]}"
crowdsec-agent-plfdn:/# cscli explain --log '2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] "GET /api/live/ws HTTP/1.1" 400 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebK
it/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"' --debug --type nginx
DEBU[24-01-2023 18:38:21] Using /etc/crowdsec/config.yaml as configuration file
line: 2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] "GET /api/live/ws HTTP/1.1" 400 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
├ s00-raw
| ├ 🔴 crowdsecurity/docker-logs
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| └ 🟢 crowdsecurity/nginx-logs (+23 ~2)
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ 🟢 crowdsecurity/geoip-enrich (+13)
| └ 🟢 crowdsecurity/http-logs (+7)
├-------- parser success 🟢
├ Scenarios
├ 🟢 crowdsecurity/http-crawl-non_statics
└ 🟢 crowdsecurity/http-probing
Content of /etc/crowdsec/acquis.yaml
---
filenames:
- /var/log/containers/*-nginx-ingress-*_*_*.log
force_inotify: true
labels:
type: docker
program: nginx