# Best way to whitelist http request

**URL:** <https://discourse.crowdsec.net/t/best-way-to-whitelist-http-request/1842>\
**Category:** crowdsec\
**Created:** [May 28, 2024, 9:37am UTC](https://discourse.crowdsec.net/t/best-way-to-whitelist-http-request/1842 "2024-05-28T09:37:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![alendroit](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@alendroit](https://discourse.crowdsec.net/u/alendroit)\
**Post date:** [May 28, 2024, 9:37am UTC](https://discourse.crowdsec.net/t/best-way-to-whitelist-http-request/1842/1 "2024-05-28T09:37:52Z")

</div>

Hello, I am looking for the best practice to whitelist http request / slug.  
I have tested multiple way but some have limitation.

1. Trough data file  
`"RegexpInFile(evt.Parsed.request, 'allowed_path.txt')"`  
where the data file is a regexp type

2. Match() string helper  
`"Match('/funcky/*/abcd*',evt.Parsed.request)"`

3. Expr  
`evt.Parsed.request matches "/funcky/.*/abcd.*"`

My problem/question are:

- With **Match()** how can I handle whithelist with character ‘?’ for example I need to whitelist: “/path?=\*” the ‘?’. How can I escape it ?
- With **Expr** I didn’t find any documentation about what it is supported in term of wildcard regex.
- The **RegexpInFile()** method can be andy but it is recommanded for whitelist ? Is this file is loaded into crowdsec after new/deleted lines ?

See you

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [May 28, 2024, 11:26am UTC](https://discourse.crowdsec.net/t/best-way-to-whitelist-http-request/1842/2 "2024-05-28T11:26:06Z")

</div>

> With **Match()** how can I handle whithelist with character ‘?’ for example I need to whitelist: “/path?=\*” the ‘?’. How can I escape it ?

Never knew the `Match` function existed till now as `matches` is more proficient in expr since it compiles to bytecode

> The **RegexpInFile()** method can be andy but it is recommanded for whitelist ? Is this file is loaded into crowdsec after new/deleted lines ?

Could be handy each line is loaded as a golang regex which has it limitations [Package regexp - The Go Programming Language](https://golang.google.cn/pkg/regexp/) the file is only loaded at startup time so realistically there no direct benefit other than yaml bloat

> With **Expr** I didn’t find any documentation about what it is supported in term of wildcard regex.

Matches is directly compiled to regex bytecode so all the support golang syntaxs can be used

---

<div class="post-metadata">

**Author:** ![alendroit](https://avatars.discourse-cdn.com/v4/letter/a/ad7895/32.png) [@alendroit](https://discourse.crowdsec.net/u/alendroit)\
**Post date:** [May 28, 2024, 2:12pm UTC](https://discourse.crowdsec.net/t/best-way-to-whitelist-http-request/1842/3 "2024-05-28T14:12:30Z")

</div>

Thanks. But how can I whitelist URL which includes “?” character ?  
I tried following:

```auto
"evt.Parsed.request matches '/data\\?path=/data/items/.*'"

```

Result:

```auto
time="2024-05-28T16:08:36+02:00" level=fatal msg="crowdsec init: while loading parsers: failed to load parser config : failed to compile node 'crowdsecurity/whitelists' in '/etc/crowdsec/parsers/s02-enrich/whitelists-test.yaml' : unable to compile whitelist expression 'evt.Parsed.request matches '/data\\?path=/data/items/.*'' : invalid char escape (1:36)\n | evt.Parsed.request matches '/data\\?path=/data/items/.*'\n | ...................................^"

```

```auto
"evt.Parsed.request matches '/data\?path=/data/items/.*'"

```

Result:

```auto
time="2024-05-28T16:10:42+02:00" level=fatal msg="while loading hub index: failed to sync items: failed to scan /etc/crowdsec: failed to unmarshal /etc/crowdsec/parsers/s02-enrich/whitelists-test.yaml: yaml: line 6: found unknown escape character"

```

```auto
evt.Parsed.request matches "/data\?path=/data/items/.*"

```

Result: Do not display error but says “parser failure”

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [May 28, 2024, 2:23pm UTC](https://discourse.crowdsec.net/t/best-way-to-whitelist-http-request/1842/4 "2024-05-28T14:23:02Z")

</div>

Yes seems the `?` does need to be escaped as tested via [Go Playground - The Go Programming Language](https://go.dev/play/p/1SyIesXw0Lc)

Since yaml is decoding you may need to escape 4 times `\\\\?` let me do some testing though

edit: can confirm double escape works