Request for comments : parsers & scenarios customization in the CrowdSec agent

[thibault]

Preface

Currently, if a CrowdSec user wants to modify a parser or a scenario, he has no choice but to edit the downloaded version, or create a complete copy. None of this options are actually convenient: items are considered tainted, and hence can’t be updated automatically.

Users have requested a better way to allow customization of one or more properties of their scenarios and parsers. This is useful in a lot of cases, for instance when you have to deal with custom web server log formats, or because you want to apply stricter/laxer policy on any given scenario etc.

The purpose of this post is to describe the approach we have in mind to add this feature to CrowdSec and collect feedback from the community.

Proposal overview

The idea is to use JavaScript Object Notation (JSON) Patch to offer a generic enough interface to allow any kind of modification to existing objects, and to wrap it into cscli tooling to make it accessible.

The proposal would allow to add a /etc/crowdsec/patches/ directory that can contain one or more yaml files. Each of these files can contain one or more patches as per below:

#I have a stricter ssh bruteforce policy
scenario: crowdsecurity/ssh-bf
patches:
  - op: replace
    path: /leakspeed
    value: 1s
---
#I'm using custom apache2 logs
parser: crowdsecurity/apache2
patches:
  - op: replace
    #/nodes/1/... targets the 2nd element of the nodes array
    path: /nodes/1/grok/pattern
    value: 'CUSTOM_PREFIX %{HTTPD_ERRORLOG}'

At startup, CrowdSec would apply all available patches on existing parsers and scenarios.

Patch creation and maintenance

To make patch creation and maintenance easier, some tooling would be included in cscli

Patch creation

cscli hub diff <file1> <file2> [-o <output>] [-a|--auto]: given that 2 files (ie. upstream scenario and locally patched copy), generate a yaml/json patch file. Displays it by default on stdout, but can write it to a given (-o) file, or to an appropriate generated file directly in /etc/crowdsec/patches/

$ cscli hub diff /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml ./my-ssh-logs.yaml

$ cscli parsers diff crowdsecurity/sshd-logs ./my-sshd-logs.yaml

Patch maintenance and preview

cscli hub patch file <patch1> <patch2> ...: given the path to an existing scenario/parser, output its final state after all patches have been applied. If paths to specific patches are given, showz the result only after these patches have been applied.

$ cscli hub patch /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml

$ cscli parsers patch crowdsecurity/sshd-logs ./mypatch1.yaml ./mypatch2.yaml

Your opinion matters to us

• Would you prefer the patch/diff feature to be all located in cscli hub or directly/as well in cscli parsers|scenarios|...?
  • Locating them in cscli hub would limit the amount of new commands
  • Locating them in cscli parsers|scenarios|... would allow to specify items by name rather than by path only (ie. cscli parsers diff crowdsecurity/sshd-logs ./myfile.yaml )

Please feel free to comment and provide general feedback,

[Sich]

If I understand (sorry it’s late) you give us 2 choices:

• Place the patch inside hub or
• directly inside the config directory.

And what about a third option where we place our patch / scenarios inside a “local” directory ? Maybe something like : /etc/crowdsec/local/config/scenarios|parsers… (for custom configuration) and /etc/crowdsec/local/patch/ to “patch” the “official” configuration ?

The idea behind that is to avoid at maximum to mix personnal files with the “official” one.

If you don’t go for the “local” folder, I would prefer to use the “hub” place to keep the “active configuration” folders “clean”.
And then in the hub folder we can use a local or custom directory like /etc/crowdsec/hub/parsers/s02-enrich/local/ .
But I don’t like too much that option because for me the “hub” directory contain the configuration get from the central repos.

I prefer to get the crowdsec/hub/ for the central repos from crowdsec.
And crowdsec/local/ for the local files.

Maybe you should use the hub/ directory if we can use a personnal repos.
Then we can store locally the files from the custom repos in /hub/parsers/s02-enrich/custom repo/

And I would like to really keep the “active configuration folders” (like crowdsec/scenarios/) to be “dynamic”, something like the “sites-enabled” in apache / nginx (at least on Debian).

I hope you understand what I mean, it’s late and the day was long

[dyllan]

I like the idea in general of the /etc/crowdsec/patches DIR to store the customisations as it conforms more to POSIX standards.

The option of storing the patch/diff feature in cscli parsers|scenarios|... etc and specifying them by name makes more sense to me, but honestly specifying the full PATH is also fine.

[dani]

This approach seems a bit complicated, but it’s probably the most flexible way. I’d have though about somthing simpler : just a parallel, partial tree in /etc/crowdsec/local/{parsers,scenarios}. All the files here would be evaluated and simply “merged on top” of the existing definitions. Eg, file /etc/crowdsec/local/scenarios/ssh-bf.yaml

---
name: crowdsecurity/ssh-bf
leakspeed: 1m
capacity: 3

Would just be merged with the definition the crowdsecurity/ssh-bf, and override leakspeed and capacity. Problem with this approach would be lists (like for nodes in parsers) where we could only override the whole thing, while the JSON patch approach would allow to do it selectively.

[athan]

Did this idea ever go anywhere? I have a custom ssh scenario simply because I want it to look over a longer time period than the crowdsecurity/sshd collection has configured. Adjusting that in an upgrade robust manner would be better than having the extra scenario.

[racitup]

Yes makes sense to have a patch for customisations, this is how fail2ban works.

[d1nd141]

I found your post because i’m also missing a feature like in fail2ban with .local file.
As a workaround, f.ex. for postfix, what i did:

i copied the /etc/crowdsec/parsers/s01-parse/postfix-logs.yaml file to the same folder as new filename: own-postfix-logs.yaml

I modified the file to remove the stock grok patterns, and added mine.
Of course, the name needs to be changed, f.ex.
name: own-postfix-logs

I did the same with scenarios:
own-nginx-errorcodes.yaml

Changed the name, capacity, leakspeed etc
Added:
scope:
type: own-nginx-rc

And as final, i changed the profiles.yaml to

---
name: enforce-own-nginx-rc
filters:
 - 'Alert.Remediation == true && Alert.GetScope() == "own-nginx-rc"'
decisions:
 - type: ban
   scope: "own-nginx-rc"
   duration: 1h
notifications:
 - email_default 
on_success: continue

[ethauvin]

Hi @thibault,

Is this still under consideration? The functionality is sorely missing, for sure.

E.

[stephdl]

same @thibault @iiAmLoz do you have found a way to alter the official scenario and parser ?

[iiAmLoz]

The status of this is still exactly the same, to prevent any overrides to scenario/parsers you still have to taint them in order to achieve it.

[dw4y]

It seems yaml files for ie the scenarios don’t work as the main config.yaml file that you can modify by creating a config.yaml.local file.

Is it planned for future ?
It could be nice if we can place *.yaml.local files in a “config.d” like directory to override the yaml of the scenario corresponding.
Anyway, it’s no so much work to redo the config when upgrade is forced and overwrite the yaml file.