Help with whitelist rules - expression with portion of URL

1

Hi,

I’ve installed crowsec on a self-hosting server I have at home. I have a https://jellyfin.org/ server there and whenever I try to access, after a few requests, the rule crowdsecurity/http-crawl-non_statics is triggered. Two things I’d like to do :

  • add whitelist to ignore URLs that have /jellyfin/ in them
  • maybe improve crowdsecurity/http-crawl-non_statics detection to not consider this as “bad traffic” (I can provide some nginx logs for this), where should I start for this ?

I’ve read https://doc.crowdsec.net/write_configurations/whitelist/ and https://doc.crowdsec.net/getting_started/concepts/#event and here is the pseudo code I though of :

  reason: "ignore jellyfin"
  expression:
     - "'/jellyfin' in evt.Parsed.request"

One of the questions I have is, how do I do a list of whitelist rules ? (I currently have the default ‘private ipv4 ranges’)

2

Hello Arthur !

Your whitelist seems correct to me (didn’t try it tho), did it work for you ?

You have a few solutions to do this :

  1. Add a new whitelist file (probably the best, so you avoid tainting existing parsers) in /etc/crowdsec/config/parsers/s02-enrich/ :
name: my-custom-whitelist
description: my custom whitelists
whitelist:
  reason: do not ban jellyfin users
  expression:
     - "'/jellyfin' in evt.Parsed.request"
  1. Add a new section to your existing whitelist file (/etc/crowdsec/config/parsers/s02-enrich/whitelists.yaml) :
name: crowdsecurity/whitelists
description: "Whitelist events from private ipv4 addresses"
whitelist:
  reason: "private ipv4 ranges"
...
---
name: my-custom-whitelist
description: my custom whitelists
whitelist:
  reason: do not ban jellyfin users
  expression:
     - "'/jellyfin' in evt.Parsed.request"
  1. Use the existing expression section of the existing whitelist :
name: crowdsecurity/whitelists
description: "Whitelist events from private ipv4 addresses"
whitelist:
  reason: "private ipv4 ranges"
  ip: 
    - "127.0.0.1"
  cidr:
    - "192.168.0.0/16"
    - "10.0.0.0/8"
    - "172.16.0.0/12"
  expression:
    - "'/jellyfin' in evt.Parsed.request"

(Please note that solutions 2 & 3 will “taint” your existing parser and it will prevent it from upgrading automatically in the future)

From what you said, crowdsecurity/http-crawl-non_statics seems to be quite prone to false positives, would you mind sharing some sample logs so we can improve it & reduce FP ?

Thanks,

3

Here is what I get when I try that expression :

time="20-10-2020 11:29:42" level=fatal msg="Unable to compile whitelist expression ''/jellyfin' in evt.Parsed.request' : invalid operation: in (mismatched types string and string) (1:13)\n | '/jellyfin' in evt.Parsed.request\n | ............^." id=twilight-snowflake name=jellyfin-whitelist stage=s02-enrich
4

Hello Arthur,

My bad, I read too fast, the in keyword is for array (X in ARRAY). To search a substring, we should use contains :

evt.Parsed.request contains '/jellyfin' (I tried it this time ^^)

Thanks,

1 Like
5

Seems to work. Thanks.

Can you tell me where the code for http-crawl-non_statics is ?

Do you have a tools to anonymise the logs before I send them ? (I think there are some tokens in the URLs)

6

Hello Arthur,

You can usually find the scenarios in /etc/crowdsec/config/scenarios/ and yours more specifically in /etc/crowdsec/config/scenarios/http-crawl-non_statics.yaml. Please note this file will be a symlink to /etc/crowdsec/config/cscli/hub/scenarios/crowdsecurity/http-crawl-non_statics.yaml.

I’m not aware of any good tool for easy anonymization of logs, and quick search only lead me to dead projects, sorry :frowning:

7

For some reason this (first option listed) turns up errors when I try it.

service crowdsec status
crowdsec.service - Crowdwatch agent
Loaded: loaded (/etc/systemd/system/crowdsec.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2020-11-07 15:43:05 GMT; 795ms ago
Process: 22442 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 30418 (code=exited, status=1/FAILURE)

Nov 07 15:43:02 xxx.com systemd[1]: Reloading Crowdwatch agent.
Nov 07 15:43:02 xxx.com systemd[1]: Reloaded Crowdwatch agent.
Nov 07 15:43:05 xxx.com systemd[1]: crowdsec.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 15:43:05 xxx.com systemd[1]: crowdsec.service: Unit entered failed state.
Nov 07 15:43:05 xxx.com systemd[1]: crowdsec.service: Failed with result ‘exit-code’.

Need to whitelist a couple static IPs which we use to access remote VPS.

Currently I’ve put the IPs in whitelists.yaml but, of course, it show up as tainted during updates.

What have I missed?

8

Hello @gppixelworks :slight_smile:

Can you show me your crafter whitelist file please ?

9

I followed the suggestions in this post but clearly missed something vital! :confounded:

Created this file at this location:

/etc/crowdsec/config/parsers/s02-enrich/mywhitelists.yaml

The contents of mywhitelists.yaml:

name: crowdsecurity/whitelists
description: "Whitelist events from my ip addresses"
whitelist:
  reason: "my ip ranges"
	ip:
		- "207.x.x.x"
10

ahah no worries, yaml can be tricky sometimes !

I think the issue is because of the indentation of your file :

name: crowdsecurity/whitelists
description: "Whitelist events from my ip addresses"
whitelist:
  reason: "my ip ranges"
  ip:
    - 207.3.4.5

As you can see here, ip needs to be at the same level as reason :slight_smile:

11

As you can see here, ip needs to be at the same level as reason

Oh for heaven sake! :grimacing:

I should have been able to figure that out.

Donkey years ago I spent a couple days troubleshooting an issue which turned out to be with a text config file. Turned out the problem was the file couldn’t end in a carriage return. Reviewing the file, I never noticed the black/blank space below the last text line. That borked the whole program.

Perfection!

Many thanks! :smiley:

1 Like