# Help with whitelist rules - expression with portion of URL

**URL:** <https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41>\
**Category:** Uncategorized\
**Created:** [October 19, 2020, 9:59am UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41 "2020-10-19T09:59:19Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![arthurlutz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/arthurlutz/32/13_2.png) [@arthurlutz](https://discourse.crowdsec.net/u/arthurlutz)\
**Post date:** [October 19, 2020, 9:59am UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/1 "2020-10-19T09:59:20Z")

</div>

Hi,

I’ve installed crowsec on a self-hosting server I have at home. I have a [https://jellyfin.org/](https://jellyfin.org/) server there and whenever I try to access, after a few requests, the rule `crowdsecurity/http-crawl-non_statics` is triggered. Two things I’d like to do :

- add whitelist to ignore URLs that have `/jellyfin/` in them
- maybe improve `crowdsecurity/http-crawl-non_statics` detection to not consider this as “bad traffic” (I can provide some nginx logs for this), where should I start for this ?

I’ve read [https://doc.crowdsec.net/write\_configurations/whitelist/](https://doc.crowdsec.net/write_configurations/whitelist/) and [https://doc.crowdsec.net/getting\_started/concepts/#event](https://doc.crowdsec.net/getting_started/concepts/#event) and here is the pseudo code I though of :

```auto
  reason: "ignore jellyfin"
  expression:
     - "'/jellyfin' in evt.Parsed.request" 

```

One of the questions I have is, how do I do a list of whitelist rules ? (I currently have the default ‘private ipv4 ranges’)

---

<div class="post-metadata">

**Author:** ![thibault](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/thibault/32/2_2.png) [@thibault](https://discourse.crowdsec.net/u/thibault)\
**Post date:** [October 19, 2020, 10:32am UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/2 "2020-10-19T10:32:40Z")

</div>

Hello Arthur !

> [@arthurlutz](#):
>
> I’ve read [CrowdSec | CrowdSec](https://doc.crowdsec.net/write_configurations/whitelist/) and [CrowdSec | CrowdSec](https://doc.crowdsec.net/getting_started/concepts/#event) and here is the pseudo code I though of :
> 
> ```auto
> reason: "ignore jellyfin"
> expression:
> - "'/jellyfin' in evt.Parsed.request" 
> 
> ```

Your whitelist seems correct to me (didn’t try it tho), did it work for you ?

> [@arthurlutz](#):
>
> One of the questions I have is, how do I do a list of whitelist rules ? (I currently have the default ‘private ipv4 ranges’)

You have a few solutions to do this :

1. Add a new whitelist file (probably the best, so you avoid tainting existing parsers) in `/etc/crowdsec/config/parsers/s02-enrich/` :

```yaml
name: my-custom-whitelist
description: my custom whitelists
whitelist:
  reason: do not ban jellyfin users
  expression:
     - "'/jellyfin' in evt.Parsed.request"

```

1. Add a new section to your existing whitelist file (`/etc/crowdsec/config/parsers/s02-enrich/whitelists.yaml`) :

```yaml
name: crowdsecurity/whitelists
description: "Whitelist events from private ipv4 addresses"
whitelist:
  reason: "private ipv4 ranges"
...
---
name: my-custom-whitelist
description: my custom whitelists
whitelist:
  reason: do not ban jellyfin users
  expression:
     - "'/jellyfin' in evt.Parsed.request"

```

1. Use the existing `expression` section of the existing whitelist :

```yaml
name: crowdsecurity/whitelists
description: "Whitelist events from private ipv4 addresses"
whitelist:
  reason: "private ipv4 ranges"
  ip: 
    - "127.0.0.1"
  cidr:
    - "192.168.0.0/16"
    - "10.0.0.0/8"
    - "172.16.0.0/12"
  expression:
    - "'/jellyfin' in evt.Parsed.request"

```

(Please note that solutions 2 & 3 will “taint” your existing parser and it will prevent it from upgrading automatically in the future)

> [@arthurlutz](#):
>
> - maybe improve `crowdsecurity/http-crawl-non_statics` detection to not consider this as “bad traffic” (I can provide some nginx logs for this), where should I start for this ?

From what you said, `crowdsecurity/http-crawl-non_statics` seems to be quite prone to false positives, would you mind sharing some sample logs so we can improve it & reduce FP ?

Thanks,

---

<div class="post-metadata">

**Author:** ![arthurlutz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/arthurlutz/32/13_2.png) [@arthurlutz](https://discourse.crowdsec.net/u/arthurlutz)\
**Post date:** [October 20, 2020, 9:30am UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/3 "2020-10-20T09:30:42Z")

</div>

Here is what I get when I try that expression :

```auto
time="20-10-2020 11:29:42" level=fatal msg="Unable to compile whitelist expression ''/jellyfin' in evt.Parsed.request' : invalid operation: in (mismatched types string and string) (1:13)\n | '/jellyfin' in evt.Parsed.request\n | ............^." id=twilight-snowflake name=jellyfin-whitelist stage=s02-enrich

```

---

<div class="post-metadata">

**Author:** ![thibault](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/thibault/32/2_2.png) [@thibault](https://discourse.crowdsec.net/u/thibault)\
**Post date:** [October 20, 2020, 12:17pm UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/4 "2020-10-20T12:17:26Z")

</div>

Hello Arthur,

> [@arthurlutz](#):
>
> Here is what I get when I try that expression :
> 
> ```auto
> time="20-10-2020 11:29:42" level=fatal msg="Unable to compile whitelist expression ''/jellyfin' in evt.Parsed.request' : invalid operation: in (mismatched types string and string) (1:13)\n | '/jellyfin' in evt.Parsed.request\n | ............^." id=twilight-snowflake name=jellyfin-whitelist stage=s02-enrich
> 
> ```

My bad, I read too fast, the `in` keyword is for array (`X in ARRAY`). To search a substring, we should use `contains` :

` evt.Parsed.request contains '/jellyfin'` (I tried it this time ^^)

Thanks,

---

<div class="post-metadata">

**Author:** ![arthurlutz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/arthurlutz/32/13_2.png) [@arthurlutz](https://discourse.crowdsec.net/u/arthurlutz)\
**Post date:** [October 20, 2020, 7:48pm UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/5 "2020-10-20T19:48:32Z")

</div>

Seems to work. Thanks.

Can you tell me where the code for `http-crawl-non_statics` is ?

Do you have a tools to anonymise the logs before I send them ? (I think there are some tokens in the URLs)

---

<div class="post-metadata">

**Author:** ![thibault](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/thibault/32/2_2.png) [@thibault](https://discourse.crowdsec.net/u/thibault)\
**Post date:** [October 21, 2020, 12:35pm UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/6 "2020-10-21T12:35:47Z")

</div>

Hello Arthur,

You can usually find the scenarios in `/etc/crowdsec/config/scenarios/` and yours more specifically in `/etc/crowdsec/config/scenarios/http-crawl-non_statics.yaml`. Please note this file will be a symlink to `/etc/crowdsec/config/cscli/hub/scenarios/crowdsecurity/http-crawl-non_statics.yaml`.

I’m not aware of any good tool for easy anonymization of logs, and quick search only lead me to dead projects, sorry ☹

---

<div class="post-metadata">

**Author:** ![gppixelworks](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/gppixelworks/32/38_2.png) [@gppixelworks](https://discourse.crowdsec.net/u/gppixelworks)\
**Post date:** [November 7, 2020, 3:53pm UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/7 "2020-11-07T15:53:18Z")

</div>

> [@thibault](#):
>
> Add a new whitelist file (probably the best, so you avoid tainting existing parsers) in `/etc/crowdsec/config/parsers/s02-enrich/` :

For some reason this (first option listed) turns up errors when I try it.

> service crowdsec status  
> crowdsec.service - Crowdwatch agent  
> Loaded: loaded (/etc/systemd/system/crowdsec.service; enabled; vendor preset: enabled)  
> Active: failed (Result: exit-code) since Sat 2020-11-07 15:43:05 GMT; 795ms ago  
> Process: 22442 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)  
> Main PID: 30418 (code=exited, status=1/FAILURE)
> 
> Nov 07 15:43:02 [xxx.com](http://xxx.com) systemd[1]: Reloading Crowdwatch agent.  
> Nov 07 15:43:02 [xxx.com](http://xxx.com) systemd[1]: Reloaded Crowdwatch agent.  
> Nov 07 15:43:05 [xxx.com](http://xxx.com) systemd[1]: crowdsec.service: Main process exited, code=exited, status=1/FAILURE  
> Nov 07 15:43:05 [xxx.com](http://xxx.com) systemd[1]: crowdsec.service: Unit entered failed state.  
> Nov 07 15:43:05 [xxx.com](http://xxx.com) systemd[1]: crowdsec.service: Failed with result ‘exit-code’.

Need to whitelist a couple static IPs which we use to access remote VPS.

Currently I’ve put the IPs in whitelists.yaml but, of course, it show up as tainted during updates.

What have I missed?

---

<div class="post-metadata">

**Author:** ![thibault](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/thibault/32/2_2.png) [@thibault](https://discourse.crowdsec.net/u/thibault)\
**Post date:** [November 10, 2020, 7:55am UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/8 "2020-11-10T07:55:06Z")

</div>

Hello @gppixelworks 🙂

Can you show me your crafter whitelist file please ?

---

<div class="post-metadata">

**Author:** ![gppixelworks](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/gppixelworks/32/38_2.png) [@gppixelworks](https://discourse.crowdsec.net/u/gppixelworks)\
**Post date:** [November 10, 2020, 10:14am UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/9 "2020-11-10T10:14:56Z")

</div>

I followed the suggestions in this post but clearly missed something vital! 😖

Created this file at this location:

`/etc/crowdsec/config/parsers/s02-enrich/mywhitelists.yaml`

The contents of mywhitelists.yaml:

```
name: crowdsecurity/whitelists
description: "Whitelist events from my ip addresses"
whitelist:
  reason: "my ip ranges"
	ip:
		- "207.x.x.x"
```

---

<div class="post-metadata">

**Author:** ![thibault](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/thibault/32/2_2.png) [@thibault](https://discourse.crowdsec.net/u/thibault)\
**Post date:** [November 10, 2020, 1:37pm UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/10 "2020-11-10T13:37:10Z")

</div>

ahah no worries, yaml can be tricky sometimes !

I think the issue is because of the indentation of your file :

```auto
name: crowdsecurity/whitelists
description: "Whitelist events from my ip addresses"
whitelist:
  reason: "my ip ranges"
  ip:
    - 207.3.4.5

```

As you can see here, `ip` needs to be at the _same_ level as reason 🙂

---

<div class="post-metadata">

**Author:** ![gppixelworks](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/gppixelworks/32/38_2.png) [@gppixelworks](https://discourse.crowdsec.net/u/gppixelworks)\
**Post date:** [November 10, 2020, 3:24pm UTC](https://discourse.crowdsec.net/t/help-with-whitelist-rules-expression-with-portion-of-url/41/11 "2020-11-10T15:24:35Z")

</div>

> As you can see here, `ip` needs to be at the _same_ level as reason

Oh for heaven sake! 😬

I should have been able to figure that out.

Donkey years ago I spent a couple days troubleshooting an issue which turned out to be with a text config file. Turned out the problem was the file couldn’t end in a carriage return. Reviewing the file, I never noticed the black/blank space below the last text line. That borked the whole program.

Perfection!

Many thanks! 😃