Help - custom parser, scenario - I think I'm close

crowdsec
1

Hello,

I’m brand new to crowdsec. I’ve got a simple host on where haproxy, zoneminder, and of course ssh are in use. There is existing functionality for haproxy and ssh. However, for zoneminder, I need to parse its web_php.log file in crowdsec for user authentication errors (i.e., bruteforce, etc.).

I’ve setup the test environment, and gone through the documentation. I think I’m very close, but not fully grokking a few things. Here’s what I’ve got setup so far:

  • added this in acquis.yml
filenames:
  - /path/to/zoneminderm/web_php.log
labels:
  type: web_php

filter: 1 == 1
debug: true
onsuccess: next_stage
name: webphp_parser
description: parser for zoneminder web_php.log
grok:
#our grok pattern : capture .*
  pattern: ^%{DATE}[- ]%{TIME}\s(%{WORD:logname})\[.*\[(%{IP:srcIP})\]\s\[%{GREEDYDATA:message}"\]
#the field to which we apply the grok pattern : the log message itself
  apply_on: message
statics:
  - parsed: web_php
    value: yes
  • added this file for my scenario
type: leaky
name: web_php/bad_logins
description: "Detect multiple bad loginsfrom a single ip"
#filter: "evt.Parsed.program  == 'web_php'"
filter: 1 == 1
groupby: "evt.Meta.srcIP"
distinct: "evt.Meta.username"
capacity: 3
leakspeed: "10s"
blackhole: 5m
labels:
 service: web_php
 type: web_php
 remediation: true

Here are some example lines of text from web_php.log:

01/06/22 09:26:11.688105 web_php[21].ERR [99.1.1.1] [Could not retrieve user testuser details] at /usr/share/zoneminder/www/includes/auth.php line 313
01/06/22 09:26:13.280121 web_php[23].ERR [99.1.1.1] [Could not retrieve user testuser details] at /usr/share/zoneminder/www/includes/auth.php line 313
01/06/22 09:26:15.117434 web_php[258].ERR [99.1.1.1] [Could not retrieve user testuser details] at /usr/share/zoneminder/www/includes/auth.php line 313
01/06/22 09:27:39.843338 web_php[688].ERR [99.1.1.1] [Login denied for user "validuser"] at /usr/share/zoneminder/www/includes/auth.php line 313
01/06/22 09:27:41.568784 web_php[25].ERR [99.1.1.1] [Login denied for user "validuser"] at /usr/share/zoneminder/www/includes/auth.php line 313

Here is my cscli metrics output:

root@zm:/etc/crowdsec/parsers# cscli metrics
INFO[06-01-2022 01:40:21 PM] Acquisition Metrics:                         
+-----------------------------------------------------+------------+--------------+----------------+------------------------+
|                       SOURCE                        | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------------------------+------------+--------------+----------------+------------------------+
| file:/path/to/zoneminderm/web_php.log               |          9 | -            |              9 | -                      |
| file:/var/log/auth.log                              |        430 | -            |            430 | -                      |
| file:/var/log/haproxy.log                           |        311 | -            |            311 | -                      |
| file:/var/log/syslog                                |        323 | -            |            323 | -                      |
| journalctl:journalctl-_SYSTEMD_UNIT=apache2.service |          2 | -            |              2 | -                      |
+-----------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[06-01-2022 01:40:21 PM] Parser Metrics:                              
+---------------+------+--------+----------+
|    PARSERS    | HITS | PARSED | UNPARSED |
+---------------+------+--------+----------+
| webphp_parser | 1075 | -      |     1075 |
+---------------+------+--------+----------+
INFO[06-01-2022 01:40:21 PM] Local Api Metrics:                           
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/decisions/stream | GET    | 2521 |
| /v1/watchers/login   | POST   |    4 |
+----------------------+--------+------+
INFO[06-01-2022 01:40:21 PM] Local Api Bouncers Metrics:                  
+----------------------------+----------------------+--------+------+
|          BOUNCER           |        ROUTE         | METHOD | HITS |
+----------------------------+----------------------+--------+------+
| FirewallBouncer-1641340869 | /v1/decisions/stream | GET    | 1260 |
| crowdsec-custom-bouncer    | /v1/decisions/stream | GET    | 1261 |
+----------------------------+----------------------+--------+------+

And getting a fair amount of errors in /var/log/crowdsec.log:

time="06-01-2022 13:39:01" level=debug msg="Event leaving node : ko" id=shy-waterfall name=webphp_parser stage=s01-parse
time="06-01-2022 13:39:01" level=debug msg="(shy-waterfall) target field 'message' doesn't exist in map[]" id=shy-waterfall name=webphp_parser stage=s01-parse
time="06-01-2022 13:39:01" level=debug msg="+ Grok '^%{DA...' didn't return data on ''" id=shy-waterfall name=webphp_parser stage=s01-parse

Here are some questions I have:

  1. In my parser, I’m not fully understanding the filter item. I had thought in the testing environment, I’d identified being able to set type type to web_php and then use this for a filter: "evt.Parsed.program == 'web_php'". However, this does not work. I’ve not found doco that shows what all of the evt.* variables are, nor to what they are mapped. Does this exist somewhere? How would I fix the filter line for my parser?

  2. In the crowdsec.log file, it’s failing to parse, even though I am able to parse in my test environment (including my filter line working). Example from test environment is below. What am I missing here?

INFO[06-01-2022 13:48:40] reading web_php3.log at once                  type="file://web_php3.log"
DEBU[06-01-2022 13:48:40] eval(evt.Parsed.program == 'web_php') = TRUE  id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] eval variables:                               id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40]        evt.Parsed.program = 'web_php'         id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] + Grok '^%{DA...' returned 3 entries to merge in Parsed  id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] 	.Parsed['logname'] = 'web_php'               id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] 	.Parsed['srcIP'] = '110.201.0.1'             id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] 	.Parsed['message'] = 'Login denied for user "hoodswoods'  id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] + Processing 1 statics                        id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] .Parsed[web_php] = 'yes'                      id=cool-frog name=webphp_parser stage=s01-parse
DEBU[06-01-2022 13:48:40] Event leaving node : ok                       id=cool-frog name=webphp_parser stage=s01-parse
  1. Do I have the right setup going here - i.e., write a parser and scenario config, and then use the crowdsec-custom-bouncer for decisions?

Thanks in advance!

-Matt

1 Like
2

Hello @tivIcar9Op,

Thank for your issue!

Here is what can i see that can be wrong:

First, you don’t seems to have the crowdsecurity/syslog-logs parser. This one is mandatory for the rest.

For your information:

  • evt.Parsed contains all the field that you have captured with your grok.
    For example with your grok, you might have evt.Parsed.logname, evt.Parsed.srcIP and evt.Parsed.message
  • evt.Meta must be populated by using statics. For example:
statics:
  - meta: source_ip
    expression: evt.Parsed.srcIP

will put the ip address that you have captured with your grok in evt.Meta.source_ip
This object is mostly use to keep important information of each logs that generate the alert when you want to view more detail about an alert (with cscli alert inspect <alert_id> -d)

In the parser:

Your filter should be :

filter: evt.Parsed.program == "web_php"

With this, the parser will only process the log from the file that you define in the acquisition file with type: web_php

This static is not needed:

statics:
  - parsed: web_php
    value: yes

The evt.Parsed.program is automatically set when you specified type: web_php in your acquisition file (in case you have the crowdsecurity/syslog-logs parser).

If you want to access evt.Meta.srcIP in your scenario filter, you should add this static in your parser:

statics:
  - meta: srcIP
    expression: evt.Parsed.srcIP

If you want to access "evt.Meta.username" in the distinct field of your scenario you should add this static in your parser:

statics:
  - meta: username
    expression: evt.Parsed.logname

In the scenario:
With this filter filter: "evt.Parsed.program == 'web_php'", all your logs from your PHP application will trigger the scenario, unless the log that you want parse only refer to a bad login.

3

Hi @alteredCoder

Thanks for the reply! I’ll ping back when I’ve had a chance to revise things.

4

@alteredCoder – looks all good now. Thank you!

Now I just need to get the rest of the pieces together.

1 Like
5

I have been told to not post questions in the discourse and also ask question in the discord sever so I am removing all my posts from discourse.

6

No, I’ve not looked into that yet. I’m still in process of getting this all setup right (working on getting the bouncer working now).

I did have to modify the grok in order to pull the right values. I came up with this:

^%{DATE}[- ]%{TIME}\s(%{WORD:logname})\[.*\[(%{IP:srcIP})\]\s\[.*\suser\s+"?(%{WORD:username})"?.*\]
7

I have been told to not post questions in the discourse and also ask question in the discord sever so I am removing all my posts from discourse.

8

Would either of you mind opening a PR on the hub ? :slight_smile:

Would be glag to see this added !

9

@baudneo - Yep… kind of a one-off at the moment. I use firewalld under the hood - and it makes for some interesting caveats (i.e., bad things happen if iptables is used and firewalld doesn’t know about it). At the moment, goal is to get this VPS rebuilt. CF is definitely on horizon.

10

@thibault – will definitely look into this as I get time. That said – it looks like @baudneo has a better handle on the parser construction than I do (I’m still learning). @baudneo - go for a PR if you like.

11

I have been told to not post questions in the discourse and also ask question in the discord sever so I am removing all my posts from discourse.