Hi @fbonalair,
Following your logs, the HTTP status seems to be on the logs now and the traefik-logs and http-logs parsers seems to parse correctly your 2 messages logs you pasted above.
Here is the log entering the traefik parser :
time="19-12-2021 12:43:07" level=debug msg="+ Grok '%{NGI...' returned 15 entries to merge in Parsed" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['traefik_server_url'] = 'http://10.42.1.239:80'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['remote_addr'] = '194.34.132.19'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['request'] = '/'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['time_local'] = '19/Dec/2021:11:43:06 +0000'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_user_agent'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['port'] = '80'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['remote_user'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_referer'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['traefik_router_name'] = 'frigg-public-routes-22735e324d6d7eb80733@kubernetescrd'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['method'] = 'GET'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['status'] = '200'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['request_duration_in_ms'] = '448'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['body_bytes_sent'] = '4646'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['number_of_requests_received_since_traefik_started'] = '138052'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_version'] = '1.1'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="move Event from stage s01-parse to s02-enrich" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="child is success, OnSuccess=next_stage, skip" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="+ Processing 7 statics" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[service] = 'http'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[http_status] = '200'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[http_path] = '/'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[user] = '-'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[source_ip] = '194.34.132.19'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[log_type] = 'http_access-log'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="setting target StrTime to 19/Dec/2021:11:43:06 +0000"
time="19-12-2021 12:43:07" level=debug msg="evt.StrTime = '19/Dec/2021:11:43:06 +0000'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="node reached the last stage : s02-enrich" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
And after that the http-logs parser
time="19-12-2021 12:43:07" level=debug msg="+ Processing 2 statics" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg=".Parsed[impact_completion] = 'true'" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="setting target Parsed.static_ressource to false"
I even tested on of your logs with cscli explain
:
$ sudo cscli explain --file /tmp/test.txt --type traefik
line: 194.34.132.19 - - [19/Dec/2021:11:43:07 +0000] "GET / HTTP/1.1" 200 4646 "-" "-" 138054 "frigg-public-routes-22735e324d6d7eb80733@kubernetescrd" "http://10.42.1.239:80" 508ms
β s00-raw
| β π΄ crowdsecurity/docker-logs
| β π’ crowdsecurity/non-syslog (first_parser)
| β π΄ crowdsecurity/syslog-logs
β s01-parse
| β π΄ crowdsecurity/nginx-logs
| β π΄ crowdsecurity/sshd-logs
| β π’ crowdsecurity/traefik-logs (+21 ~2)
β s02-enrich
| β π’ crowdsecurity/dateparse-enrich (+1 ~1)
| β π’ crowdsecurity/geoip-enrich (+9)
| β π΄ crowdsecurity/http-logs
| β π’ crowdsecurity/whitelists (+2)
β-------- parser success π’
β Scenarios
β π’ crowdsecurity/http-crawl-non_statics
Maybe if you try to run cscli explain
and see whatβs happen (how many logs are parsed, by which parsers etc β¦)