# Request for comments : parsers & scenarios customization in the CrowdSec agent

**URL:** <https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129>\
**Category:** crowdsec\
**Created:** [March 4, 2021, 12:28pm UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129 "2021-03-04T12:28:59Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![thibault](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/thibault/32/2_2.png) [@thibault](https://discourse.crowdsec.net/u/thibault)\
**Post date:** [March 4, 2021, 12:28pm UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/1 "2021-03-04T12:28:59Z")

</div>

## Preface

Currently, if a CrowdSec user wants to modify a parser or a scenario, he has no choice but to edit the downloaded version, or create a complete copy. None of this options are actually convenient: items are considered tainted, and hence can’t be updated automatically.

Users have requested a better way to allow customization of one or more properties of their scenarios and parsers. This is useful in a lot of cases, for instance when you have to deal with custom web server log formats, or because you want to apply stricter/laxer policy on any given scenario etc.

The purpose of this post is to describe the approach we have in mind to add this feature to CrowdSec and collect feedback from the community.

## Proposal overview

The idea is to use [JavaScript Object Notation (JSON) Patch](https://tools.ietf.org/html/rfc6902) to offer a generic enough interface to allow any kind of modification to existing objects, and to wrap it into `cscli` tooling to make it accessible.

The proposal would allow to add a `/etc/crowdsec/patches/` directory that can contain one or more yaml files. Each of these files can contain one or more patches as per below:

```auto
#I have a stricter ssh bruteforce policy
scenario: crowdsecurity/ssh-bf
patches:
  - op: replace
    path: /leakspeed
    value: 1s
---
#I'm using custom apache2 logs
parser: crowdsecurity/apache2
patches:
  - op: replace
    #/nodes/1/... targets the 2nd element of the nodes array
    path: /nodes/1/grok/pattern
    value: 'CUSTOM_PREFIX %{HTTPD_ERRORLOG}'

```

At startup, CrowdSec would apply all available patches on existing parsers and scenarios.

## Patch creation and maintenance

To make patch creation and maintenance easier, some tooling would be included in `cscli`

### Patch creation

`cscli hub diff <file1> <file2> [-o <output>] [-a|--auto]`: given that 2 files (ie. upstream scenario and locally patched copy), generate a yaml/json patch file. Displays it by default on stdout, but can write it to a given (`-o`) file, or to an appropriate generated file directly in `/etc/crowdsec/patches/`

```bash
$ cscli hub diff /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml ./my-ssh-logs.yaml

$ cscli parsers diff crowdsecurity/sshd-logs ./my-sshd-logs.yaml

```

### Patch maintenance and preview

`cscli hub patch file <patch1> <patch2> ...`: given the path to an existing scenario/parser, output its final state after all patches have been applied. If paths to specific patches are given, showz the result **only** after these patches have been applied.

```nohighlight
$ cscli hub patch /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml

$ cscli parsers patch crowdsecurity/sshd-logs ./mypatch1.yaml ./mypatch2.yaml

```

### Your opinion matters to us

- Would you prefer the patch/diff feature to be all located in `cscli hub` or directly/as well in `cscli parsers|scenarios|...`?
  - Locating them in `cscli hub` would limit the amount of new commands
  - Locating them in `cscli parsers|scenarios|...` would allow to specify items by name rather than by path only (ie. `cscli parsers diff crowdsecurity/sshd-logs ./myfile.yaml` )

Please feel free to comment and provide general feedback,

---

<div class="post-metadata">

**Author:** ![thibault](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/thibault/32/2_2.png) [@thibault](https://discourse.crowdsec.net/u/thibault)\
**Post date:** [March 4, 2021, 4:31pm UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/2 "2021-03-04T16:31:17Z")

</div>



---

<div class="post-metadata">

**Author:** ![Sich](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/sich/32/16_2.png) [@Sich](https://discourse.crowdsec.net/u/Sich)\
**Post date:** [March 4, 2021, 7:15pm UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/3 "2021-03-04T19:15:46Z")

</div>

If I understand (sorry it’s late) you give us 2 choices:

- Place the patch inside hub or
- directly inside the config directory.

And what about a third option where we place our patch / scenarios inside a “local” directory ? Maybe something like : /etc/crowdsec/local/config/scenarios|parsers… (for custom configuration) and /etc/crowdsec/local/patch/ to “patch” the “official” configuration ?

The idea behind that is to avoid at maximum to mix personnal files with the “official” one.

If you don’t go for the “local” folder, I would prefer to use the “hub” place to keep the “active configuration” folders “clean”.  
And then in the hub folder we can use a local or custom directory like /etc/crowdsec/hub/parsers/s02-enrich/local/ .  
But I don’t like too much that option because for me the “hub” directory contain the configuration get from the central repos.

I prefer to get the crowdsec/hub/ for the central repos from crowdsec.  
And crowdsec/local/ for the local files.

Maybe you should use the hub/ directory if we can use a personnal repos.  
Then we can store locally the files from the custom repos in /hub/parsers/s02-enrich/custom repo/

And I would like to really keep the “active configuration folders” (like crowdsec/scenarios/) to be “dynamic”, something like the “sites-enabled” in apache / nginx (at least on Debian).

I hope you understand what I mean, it’s late and the day was long 🙂

---

<div class="post-metadata">

**Author:** ![dyllan](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/dyllan/32/83_2.png) [@dyllan](https://discourse.crowdsec.net/u/dyllan)\
**Post date:** [March 5, 2021, 3:26am UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/4 "2021-03-05T03:26:35Z")

</div>

I like the idea in general of the `/etc/crowdsec/patches` DIR to store the customisations as it conforms more to POSIX standards.

The option of storing the patch/diff feature in `cscli parsers|scenarios|...` etc and specifying them by name makes more sense to me, but honestly specifying the full PATH is also fine.

---

<div class="post-metadata">

**Author:** ![dani](https://avatars.discourse-cdn.com/v4/letter/d/35a633/32.png) [@dani](https://discourse.crowdsec.net/u/dani)\
**Post date:** [March 5, 2021, 7:35am UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/5 "2021-03-05T07:35:01Z")

</div>

This approach seems a bit complicated, but it’s probably the most flexible way. I’d have though about somthing simpler : just a parallel, partial tree in /etc/crowdsec/local/{parsers,scenarios}. All the files here would be evaluated and simply “merged on top” of the existing definitions. Eg, file /etc/crowdsec/local/scenarios/ssh-bf.yaml

```auto
---
name: crowdsecurity/ssh-bf
leakspeed: 1m
capacity: 3

```

Would just be merged with the definition the crowdsecurity/ssh-bf, and override leakspeed and capacity. Problem with this approach would be lists (like for nodes in parsers) where we could only override the whole thing, while the JSON patch approach would allow to do it selectively.

---

<div class="post-metadata">

**Author:** ![athan](https://avatars.discourse-cdn.com/v4/letter/a/f475e1/32.png) [@athan](https://discourse.crowdsec.net/u/athan)\
**Post date:** [February 19, 2023, 11:37am UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/6 "2023-02-19T11:37:19Z")

</div>

Did this idea ever go anywhere? I have a custom ssh scenario simply because I want it to look over a longer time period than the crowdsecurity/sshd collection has configured. Adjusting that in an upgrade robust manner would be better than having the extra scenario.

---

<div class="post-metadata">

**Author:** ![racitup](https://avatars.discourse-cdn.com/v4/letter/r/c37758/32.png) [@racitup](https://discourse.crowdsec.net/u/racitup)\
**Post date:** [February 28, 2023, 4:19pm UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/7 "2023-02-28T16:19:12Z")

</div>

Yes makes sense to have a patch for customisations, this is how fail2ban works.

---

<div class="post-metadata">

**Author:** ![d1nd141](https://avatars.discourse-cdn.com/v4/letter/d/cab0a1/32.png) [@d1nd141](https://discourse.crowdsec.net/u/d1nd141)\
**Post date:** [August 16, 2023, 3:11pm UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/8 "2023-08-16T15:11:17Z")

</div>

I found your post because i’m also missing a feature like in fail2ban with .local file.  
As a workaround, f.ex. for postfix, what i did:

i copied the /etc/crowdsec/parsers/s01-parse/postfix-logs.yaml file to the same folder as new filename: own-postfix-logs.yaml

I modified the file to remove the stock grok patterns, and added mine.  
Of course, the name needs to be changed, f.ex.  
name: own-postfix-logs

I did the same with scenarios:  
own-nginx-errorcodes.yaml

Changed the name, capacity, leakspeed etc  
Added:  
scope:  
type: own-nginx-rc

And as final, i changed the profiles.yaml to

```auto
---
name: enforce-own-nginx-rc
filters:
 - 'Alert.Remediation == true && Alert.GetScope() == "own-nginx-rc"'
decisions:
 - type: ban
   scope: "own-nginx-rc"
   duration: 1h
notifications:
 - email_default 
on_success: continue

```

---

<div class="post-metadata">

**Author:** ![ethauvin](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/ethauvin/32/668_2.png) [@ethauvin](https://discourse.crowdsec.net/u/ethauvin)\
**Post date:** [September 17, 2023, 6:16pm UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/9 "2023-09-17T18:16:30Z")

</div>

Hi @thibault,

Is this still under consideration? The functionality is sorely missing, for sure.

E.

---

<div class="post-metadata">

**Author:** ![stephdl](https://avatars.discourse-cdn.com/v4/letter/s/439d5e/32.png) [@stephdl](https://discourse.crowdsec.net/u/stephdl)\
**Post date:** [September 10, 2024, 6:37am UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/10 "2024-09-10T06:37:50Z")

</div>

same @thibault @iiAmLoz do you have found a way to alter the official scenario and parser ?

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [September 10, 2024, 8:57am UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/11 "2024-09-10T08:57:10Z")

</div>

The status of this is still exactly the same, to prevent any overrides to scenario/parsers you still have to taint them in order to achieve it.

---

<div class="post-metadata">

**Author:** ![dw4y](https://avatars.discourse-cdn.com/v4/letter/d/cab0a1/32.png) [@dw4y](https://discourse.crowdsec.net/u/dw4y)\
**Post date:** [December 10, 2024, 11:18pm UTC](https://discourse.crowdsec.net/t/request-for-comments-parsers-scenarios-customization-in-the-crowdsec-agent/129/12 "2024-12-10T23:18:36Z")

</div>

It seems yaml files for ie the scenarios don’t work as the main config.yaml file that you can modify by creating a config.yaml.local file.

Is it planned for future ?  
It could be nice if we can place \*.yaml.local files in a “config.d” like directory to override the yaml of the scenario corresponding.  
Anyway, it’s no so much work to redo the config when upgrade is forced and overwrite the yaml file.