Best way to whitelist http request

crowdsec
1

Hello, I am looking for the best practice to whitelist http request / slug.
I have tested multiple way but some have limitation.

  1. Trough data file
    "RegexpInFile(evt.Parsed.request, 'allowed_path.txt')"
    where the data file is a regexp type

  2. Match() string helper
    "Match('/funcky/*/abcd*',evt.Parsed.request)"

  3. Expr
    evt.Parsed.request matches "/funcky/.*/abcd.*"

My problem/question are:

  • With Match() how can I handle whithelist with character ‘?’ for example I need to whitelist: “/path?=*” the ‘?’. How can I escape it ?
  • With Expr I didn’t find any documentation about what it is supported in term of wildcard regex.
  • The RegexpInFile() method can be andy but it is recommanded for whitelist ? Is this file is loaded into crowdsec after new/deleted lines ?

See you

2

With Match() how can I handle whithelist with character ‘?’ for example I need to whitelist: “/path?=*” the ‘?’. How can I escape it ?

Never knew the Match function existed till now as matches is more proficient in expr since it compiles to bytecode

The RegexpInFile() method can be andy but it is recommanded for whitelist ? Is this file is loaded into crowdsec after new/deleted lines ?

Could be handy each line is loaded as a golang regex which has it limitations Package regexp - The Go Programming Language the file is only loaded at startup time so realistically there no direct benefit other than yaml bloat

With Expr I didn’t find any documentation about what it is supported in term of wildcard regex.

Matches is directly compiled to regex bytecode so all the support golang syntaxs can be used

3

Thanks. But how can I whitelist URL which includes “?” character ?
I tried following:

"evt.Parsed.request matches '/data\\?path=/data/items/.*'"

Result:

time="2024-05-28T16:08:36+02:00" level=fatal msg="crowdsec init: while loading parsers: failed to load parser config : failed to compile node 'crowdsecurity/whitelists' in '/etc/crowdsec/parsers/s02-enrich/whitelists-test.yaml' : unable to compile whitelist expression 'evt.Parsed.request matches '/data\\?path=/data/items/.*'' : invalid char escape (1:36)\n | evt.Parsed.request matches '/data\\?path=/data/items/.*'\n | ...................................^"

"evt.Parsed.request matches '/data\?path=/data/items/.*'"

Result:

time="2024-05-28T16:10:42+02:00" level=fatal msg="while loading hub index: failed to sync items: failed to scan /etc/crowdsec: failed to unmarshal /etc/crowdsec/parsers/s02-enrich/whitelists-test.yaml: yaml: line 6: found unknown escape character"

evt.Parsed.request matches "/data\?path=/data/items/.*"

Result: Do not display error but says “parser failure”

4

Yes seems the ? does need to be escaped as tested via Go Playground - The Go Programming Language

Since yaml is decoding you may need to escape 4 times \\\\? let me do some testing though

edit: can confirm double escape works