Agent fails to parse logs - but cscli explain works

joaogbcravo · January 25, 2023, 10:29am

On a kubernetes cluster, everything looks working fine except agents fails parsing nginx logs.

Running cscli explain inside the agent container shows no errors.

What can it be?

Agent logs:

time="24-01-2023 18:37:16" level=debug msg="pushing {Raw:2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] \"GET /api/live/ws HTTP/1.1\" 400 12 \"-\" \"Mozilla/5.0 (X11; Linux
x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36\" \"-\" Src:/var/log/containers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34c
bf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log Time:2023-01-24 18:37:16.203161076 +0000 UTC m=+2349.726670066 Labels:map[program:nginx type:docker] Process:true Module:file}" tail=/var/log/cont
ainers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34cbf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log type=file
time="24-01-2023 18:37:16" level=debug msg="+ Processing 3 statics" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="[time] doesn't exist"
time="24-01-2023 18:37:16" level=debug msg="Empty value for evt.StrTime, skip." id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="[log] doesn't exist"
time="24-01-2023 18:37:16" level=debug msg="Empty value for .Parsed[message], skip." id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg=".Parsed[program] = 'nginx'" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ok" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="move Event from stage s00-raw to s01-parse" id=black-leaf name=crowdsecurity/docker-logs stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="node successful, stop end stage s00-raw" node-name=black-leaf stage=s00-raw
time="24-01-2023 18:37:16" level=debug msg="(black-snow) target field 'message' doesn't exist in map[program:nginx]" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="+ Grok '(%{IP...' didn't return data on ''" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=black-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="(wandering-snow) target field 'message' doesn't exist in map[program:nginx]" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="+ Grok '(%{IP...' didn't return data on ''" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=falling-thunder name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=late-moon name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=divine-shadow name=child-child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=wandering-snow name=child-crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko" id=little-water name=crowdsecurity/nginx-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Event leaving node : ko (failed filter)" id=hidden-sky name=crowdsecurity/sshd-logs stage=s01-parse
time="24-01-2023 18:37:16" level=debug msg="Log didn't finish stage s01-parse"
time="24-01-2023 18:37:16" level=debug msg="Discarding line {Type:0 ExpectMode:0 Whitelisted:false WhitelistReason: Stage:s01-parse Line:{Raw:2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] \"GET /api/live/ws HTTP/1.1\" 400 12 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36\" \"-\" Src:/var/log/containers/nginx-ingress-controller-nginx-ingress-7fff685d4d-5b49w_ops_nginx-ingress-controller-nginx-ingress-34cbf559436917597109f2418374864fefaeabf1e4d0a743458566abf625bf9a.log Time:2023-01-24 18:37:16.203161076 +0000 UTC m=+2349.726670066 Labels:map[program:nginx type:docker] Process:true Module:file} Parsed:map[program:nginx] Enriched:map[] Overflow:{Mapkey: BucketId: Whitelisted:false Reprocess:false Sources:map[] Alert:<nil> APIAlerts:[]} Time:2023-01-24 18:37:16.203255925 +0000 UTC StrTime: StrTimeFormat: MarshaledTime: Process:false Meta:map[]}"

cscli explain

cscli explain --log '2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] "GET /api/live/ws HTTP/1.1" 400 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"' --debug --type nginx
DEBU[24-01-2023 18:38:21] Using /etc/crowdsec/config.yaml as configuration file
line: 2023-01-24T18:37:16.031084452Z stdout F 213.3.24.157 - - [24/Jan/2023:18:37:16 +0000] "GET /api/live/ws HTTP/1.1" 400 12 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/docker-logs
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       └ 🟢 crowdsecurity/nginx-logs (+23 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+13)
        |       └ 🟢 crowdsecurity/http-logs (+7)
        ├-------- parser success 🟢
        ├ Scenarios
                ├ 🟢 crowdsecurity/http-crawl-non_statics
                └ 🟢 crowdsecurity/http-probing

Content of /etc/crowdsec/acquis.yaml

filenames:
  - /var/log/containers/*-nginx-ingress-*_*_*.log
force_inotify: true
labels:
  type: docker
  program: nginx

joaogbcravo · January 26, 2023, 12:40pm

The problem is related with Crowdsec agent cannot parse Nginx ingress logs unless "container_runtime: nginx" · Issue #66 · crowdsecurity/helm-charts · GitHub

joaogbcravo · January 26, 2023, 1:42pm

For future travelers, if you are running on kubernetes and your runtime is not docker, you need to setup the correct type here. In my case was containerd.

If you are using helm, make sure to override this variable on your values.yaml.

github.com

crowdsecurity/helm-charts/blob/20f64df00a413e037ee2ca63afd9e9dc7ee9b393/charts/crowdsec/values.yaml#L8


      
          # yaml-language-server: $schema=https://raw.githubusercontent.com/crowdsecurity/helm-charts/main/charts/crowdsec/values.schema.json
          
          # Default values for crowdsec-chart.
          # This is a YAML-formatted file.
          # Declare variables to be passed into your templates.
          
          # -- for raw logs format: json or cri (docker|containerd)
          container_runtime: docker
          
          image:
            # -- docker image repository name
            repository: crowdsecurity/crowdsec
            # -- pullPolicy
            pullPolicy: IfNotPresent
            # -- docker image tag
            tag: ""
          
          # Here you can specify your own custom configuration to be loaded in crowdsec agent or lapi

Topic		Replies	Views
Nginx logs not being parsed - cscli explain works crowdsec	1	973	January 31, 2023
Ingress Nginx parse file on One agent crowdsec	2	331	November 2, 2023
Nginx logs not being parsed	6	1351	September 15, 2021
Solution for parsing logs of docker containers crowdsec	5	4196	July 30, 2022
Crowdsec failing to start after executing cscli explain crowdsec	2	1765	May 25, 2022

Agent fails to parse logs - but cscli explain works

Related topics