# Help - custom parser, scenario - I think I'm close

**URL:** <https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513>\
**Category:** crowdsec\
**Created:** [January 6, 2022, 7:57pm UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513 "2022-01-06T19:57:34Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![tivIcar9Op](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tivIcar9Op](https://discourse.crowdsec.net/u/tivIcar9Op)\
**Post date:** [January 6, 2022, 7:57pm UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/1 "2022-01-06T19:57:34Z")

</div>

Hello,

I’m brand new to crowdsec. I’ve got a simple host on where haproxy, zoneminder, and of course ssh are in use. There is existing functionality for haproxy and ssh. However, for zoneminder, I need to parse its web\_php.log file in crowdsec for user authentication errors (i.e., bruteforce, etc.).

I’ve setup the test environment, and gone through the documentation. I think I’m very close, but not fully grokking a few things. Here’s what I’ve got setup so far:

- added this in acquis.yml

```auto
filenames:
  - /path/to/zoneminderm/web_php.log
labels:
  type: web_php

```

- set this up for my custom parser (and used [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) to validate):

```auto
filter: 1 == 1
debug: true
onsuccess: next_stage
name: webphp_parser
description: parser for zoneminder web_php.log
grok:
#our grok pattern : capture .*
  pattern: ^%{DATE}[-]%{TIME}\s(%{WORD:logname})\[.*\[(%{IP:srcIP})\]\s\[%{GREEDYDATA:message}"\]
#the field to which we apply the grok pattern : the log message itself
  apply_on: message
statics:
  - parsed: web_php
    value: yes

```

- added this file for my scenario

```auto
type: leaky
name: web_php/bad_logins
description: "Detect multiple bad loginsfrom a single ip"
#filter: "evt.Parsed.program == 'web_php'"
filter: 1 == 1
groupby: "evt.Meta.srcIP"
distinct: "evt.Meta.username"
capacity: 3
leakspeed: "10s"
blackhole: 5m
labels:
 service: web_php
 type: web_php
 remediation: true

```

Here are some example lines of text from web\_php.log:

```auto
01/06/22 09:26:11.688105 web_php[21].ERR [99.1.1.1] [Could not retrieve user testuser details] at /usr/share/zoneminder/www/includes/auth.php line 313
01/06/22 09:26:13.280121 web_php[23].ERR [99.1.1.1] [Could not retrieve user testuser details] at /usr/share/zoneminder/www/includes/auth.php line 313
01/06/22 09:26:15.117434 web_php[258].ERR [99.1.1.1] [Could not retrieve user testuser details] at /usr/share/zoneminder/www/includes/auth.php line 313
01/06/22 09:27:39.843338 web_php[688].ERR [99.1.1.1] [Login denied for user "validuser"] at /usr/share/zoneminder/www/includes/auth.php line 313
01/06/22 09:27:41.568784 web_php[25].ERR [99.1.1.1] [Login denied for user "validuser"] at /usr/share/zoneminder/www/includes/auth.php line 313

```

Here is my cscli metrics output:

```auto
root@zm:/etc/crowdsec/parsers# cscli metrics
INFO[06-01-2022 01:40:21 PM] Acquisition Metrics:                         
+-----------------------------------------------------+------------+--------------+----------------+------------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------------------------+------------+--------------+----------------+------------------------+
| file:/path/to/zoneminderm/web_php.log | 9 | - | 9 | - |
| file:/var/log/auth.log | 430 | - | 430 | - |
| file:/var/log/haproxy.log | 311 | - | 311 | - |
| file:/var/log/syslog | 323 | - | 323 | - |
| journalctl:journalctl-_SYSTEMD_UNIT=apache2.service | 2 | - | 2 | - |
+-----------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[06-01-2022 01:40:21 PM] Parser Metrics:                              
+---------------+------+--------+----------+
| PARSERS | HITS | PARSED | UNPARSED |
+---------------+------+--------+----------+
| webphp_parser | 1075 | - | 1075 |
+---------------+------+--------+----------+
INFO[06-01-2022 01:40:21 PM] Local Api Metrics:                           
+----------------------+--------+------+
| ROUTE | METHOD | HITS |
+----------------------+--------+------+
| /v1/decisions/stream | GET | 2521 |
| /v1/watchers/login | POST | 4 |
+----------------------+--------+------+
INFO[06-01-2022 01:40:21 PM] Local Api Bouncers Metrics:                  
+----------------------------+----------------------+--------+------+
| BOUNCER | ROUTE | METHOD | HITS |
+----------------------------+----------------------+--------+------+
| FirewallBouncer-1641340869 | /v1/decisions/stream | GET | 1260 |
| crowdsec-custom-bouncer | /v1/decisions/stream | GET | 1261 |
+----------------------------+----------------------+--------+------+

```

And getting a fair amount of errors in /var/log/crowdsec.log:

```auto
time="06-01-2022 13:39:01" level=debug msg="Event leaving node : ko" id=shy-waterfall name=webphp_parser stage=s01-parse
time="06-01-2022 13:39:01" level=debug msg="(shy-waterfall) target field 'message' doesn't exist in map[]" id=shy-waterfall name=webphp_parser stage=s01-parse
time="06-01-2022 13:39:01" level=debug msg="+ Grok '^%{DA...' didn't return data on ''" id=shy-waterfall name=webphp_parser stage=s01-parse

```

Here are some questions I have:

1. In my parser, I’m not fully understanding the filter item. I had thought in the testing environment, I’d identified being able to set type type to `web_php` and then use this for a filter: `"evt.Parsed.program == 'web_php'"`. However, this does not work. I’ve not found doco that shows what all of the evt.\* variables are, nor to what they are mapped. Does this exist somewhere? How would I fix the filter line for my parser?

2. In the crowdsec.log file, it’s failing to parse, even though I am able to parse in my test environment (including my filter line working). Example from test environment is below. What am I missing here?

```auto
INFO[06-01-2022 13:48:40] reading web_php3.log at once type="file://web_php3.log"
DEBU[06-01-2022 13:48:40] eval(evt.Parsed.program == 'web_php') = TRUE id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] eval variables: id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] evt.Parsed.program = 'web_php' id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] + Grok '^%{DA...' returned 3 entries to merge in Parsed id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] .Parsed['logname'] = 'web_php' id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] .Parsed['srcIP'] = '110.201.0.1' id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] .Parsed['message'] = 'Login denied for user "hoodswoods' id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] + Processing 1 statics id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] .Parsed[web_php] = 'yes' id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] Event leaving node : ok id=cool-frog name=webphp_parser stage=s01-parse

```

1. Do I have the right setup going here - i.e., write a parser and scenario config, and then use the crowdsec-custom-bouncer for decisions?

Thanks in advance!

-Matt

---

<div class="post-metadata">

**Author:** ![alteredCoder](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/alteredcoder/32/44_2.png) [@alteredCoder](https://discourse.crowdsec.net/u/alteredCoder)\
**Post date:** [January 7, 2022, 11:49am UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/2 "2022-01-07T11:49:59Z")

</div>

Hello @tivIcar9Op,

Thank for your issue!

Here is what can i see that can be wrong:

First, you don’t seems to have the `crowdsecurity/syslog-logs` parser. This one is mandatory for the rest.

For your information:

- `evt.Parsed` contains all the field that you have captured with your grok.  
For example with your grok, you might have `evt.Parsed.logname`, `evt.Parsed.srcIP` and `evt.Parsed.message`
- `evt.Meta` must be populated by using statics. For example:

```auto
statics:
  - meta: source_ip
    expression: evt.Parsed.srcIP

```

will put the ip address that you have captured with your grok in `evt.Meta.source_ip`  
This object is mostly use to keep important information of each logs that generate the alert when you want to view more detail about an alert (with `cscli alert inspect <alert_id> -d`)

In the parser:

Your filter should be :

```auto
filter: evt.Parsed.program == "web_php"

```

With this, the parser will only process the log from the file that you define in the acquisition file with `type: web_php`

This static is not needed:

```auto
statics:
  - parsed: web_php
    value: yes

```

The `evt.Parsed.program` is automatically set when you specified `type: web_php` in your acquisition file (in case you have the `crowdsecurity/syslog-logs` parser).

If you want to access `evt.Meta.srcIP` in your scenario filter, you should add this static in your parser:

```auto
statics:
  - meta: srcIP
    expression: evt.Parsed.srcIP

```

If you want to access `"evt.Meta.username"` in the distinct field of your scenario you should add this static in your parser:

```auto
statics:
  - meta: username
    expression: evt.Parsed.logname

```

In the scenario:  
With this filter `filter: "evt.Parsed.program == 'web_php'"`, all your logs from your PHP application will trigger the scenario, unless the log that you want parse only refer to a bad login.

---

<div class="post-metadata">

**Author:** ![tivIcar9Op](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tivIcar9Op](https://discourse.crowdsec.net/u/tivIcar9Op)\
**Post date:** [January 7, 2022, 6:58pm UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/3 "2022-01-07T18:58:06Z")

</div>

Hi @alteredCoder

Thanks for the reply! I’ll ping back when I’ve had a chance to revise things.

---

<div class="post-metadata">

**Author:** ![tivIcar9Op](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tivIcar9Op](https://discourse.crowdsec.net/u/tivIcar9Op)\
**Post date:** [January 18, 2022, 5:46pm UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/4 "2022-01-18T17:46:19Z")

</div>

@alteredCoder – looks all good now. Thank you!

Now I just need to get the rest of the pieces together.

---

<div class="post-metadata">

**Author:** ![baudneo](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/baudneo/32/186_2.png) [@baudneo](https://discourse.crowdsec.net/u/baudneo)\
**Post date:** [January 24, 2022, 10:01pm UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/5 "2022-01-24T22:01:15Z")

</div>

I have been told to not post questions in the discourse and also ask question in the discord sever so I am removing all my posts from discourse.

---

<div class="post-metadata">

**Author:** ![tivIcar9Op](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tivIcar9Op](https://discourse.crowdsec.net/u/tivIcar9Op)\
**Post date:** [January 24, 2022, 11:59pm UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/6 "2022-01-24T23:59:45Z")

</div>

No, I’ve not looked into that yet. I’m still in process of getting this all setup right (working on getting the bouncer working now).

I did have to modify the grok in order to pull the right values. I came up with this:

```auto
^%{DATE}[-]%{TIME}\s(%{WORD:logname})\[.*\[(%{IP:srcIP})\]\s\[.*\suser\s+"?(%{WORD:username})"?.*\]

```

---

<div class="post-metadata">

**Author:** ![baudneo](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/baudneo/32/186_2.png) [@baudneo](https://discourse.crowdsec.net/u/baudneo)\
**Post date:** [January 25, 2022, 12:19am UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/7 "2022-01-25T00:19:29Z")

</div>

I have been told to not post questions in the discourse and also ask question in the discord sever so I am removing all my posts from discourse.

---

<div class="post-metadata">

**Author:** ![thibault](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/thibault/32/2_2.png) [@thibault](https://discourse.crowdsec.net/u/thibault)\
**Post date:** [January 25, 2022, 8:56am UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/8 "2022-01-25T08:56:43Z")

</div>

Would either of you mind opening a PR on the hub ? 🙂

Would be glag to see this added !

---

<div class="post-metadata">

**Author:** ![tivIcar9Op](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tivIcar9Op](https://discourse.crowdsec.net/u/tivIcar9Op)\
**Post date:** [January 26, 2022, 4:08pm UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/9 "2022-01-26T16:08:54Z")

</div>

@baudneo - Yep… kind of a one-off at the moment. I use firewalld under the hood - and it makes for some interesting caveats (i.e., bad things happen if iptables is used and firewalld doesn’t know about it). At the moment, goal is to get this VPS rebuilt. CF is definitely on horizon.

---

<div class="post-metadata">

**Author:** ![tivIcar9Op](https://avatars.discourse-cdn.com/v4/letter/t/58956e/32.png) [@tivIcar9Op](https://discourse.crowdsec.net/u/tivIcar9Op)\
**Post date:** [January 26, 2022, 4:10pm UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/10 "2022-01-26T16:10:27Z")

</div>

@thibault – will definitely look into this as I get time. That said – it looks like @baudneo has a better handle on the parser construction than I do (I’m still learning). @baudneo - go for a PR if you like.

---

<div class="post-metadata">

**Author:** ![baudneo](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/baudneo/32/186_2.png) [@baudneo](https://discourse.crowdsec.net/u/baudneo)\
**Post date:** [February 6, 2022, 11:40am UTC](https://discourse.crowdsec.net/t/help-custom-parser-scenario-i-think-im-close/513/11 "2022-02-06T11:40:21Z")

</div>

I have been told to not post questions in the discourse and also ask question in the discord sever so I am removing all my posts from discourse.