Watching logs for errors

1

Hello,

I’m totally new to CrowdSec but read nearly all the documentation. So I have the basic knowledge and have been using OSSEC for at least a decade. I’m asking help on how to do the following with CrowdSec:

  1. Watch all the messages coming trough journald. That is done, as I filtered the journal via _HOSTNAME=myhostname in acquis.yaml
  2. Catch those journal messages which have the word “error” in the message anywhere, any unit…
  3. There is no need for user names or IP addresses to be extracted from the log line, no remediation, no bucketing.
  4. Notification should be sent containing the raw line.

I understand that CrowdSec is not a monitoring or alerting solution, but I need what CrowdSec does plus this, so I wouldn’t like to install some other software just for tasks like the above one.

Any help, any direction is appreciated. Thanks in advance.

2

Yes it doable here a step by step guide:

1: Setup acquisition (You may have already done this)

source: journalctl
journalctl_filter:
 - "_HOSTNAME=myhostname"
labels:
  type: syslog
  onlyerrors: ""

Key thing to point out we want to create an extra label to make sure we filter other journalctl sources out (I know this might be your only one but want to make sure if anyone finds this post it will work too)

2: Setup a s01-parser to filter and only find valid error lines

## /etc/crowdsec/s01-parse/journald-errors.yaml
onsuccess: next_stage
debug: false
filter: "'onlyerrors' in evt.Line.Labels && evt.Line.Module == 'journalctl' && Lower(evt.Line.Raw) contains 'error'"
name: me/jounalctl-errors
description: "Parse error logs"
statics:
  - meta: log_type
    value: journalctl-error
## We assign the raw line to Meta attribute so we can get access in notifications
  - meta: raw
    expression: "evt.Line.Raw"

Now once that is complete we have to create a trigger bucket so every single even will generate an alert. (Later in notifications we will use alert grouping if you want to stop spamming thats if you want)

3: Trigger bucket scenario

## /etc/crowdsec/scenarios/journalctl-error.yaml
type: trigger
name: me/journalctl-error
description: "Trigger bucket for single journalctl error"
filter: "evt.Meta.log_type == 'journalctl-error'"
capacity: 0
labels:
  remediation: false
scope:
  type: error
  expression: "1"

4: Profiles catch to alert

name: journalctl-error
filters:
  - Alert.Remediation == false && Alert.GetScenario() == "me/journalctl-error"
notifications:
  - slack_default
on_success: break
## VERY IMPORTANT LINE YOU NEED TO BREAK HERE ##
---
name: default_ip_remediation
#debug: true
filters:
 - Alert.Remediation == true && Alert.GetScope() == "Ip"
decisions:
 - type: ban
   duration: 4h
#duration_expr: "Sprintf('%dh', (GetDecisionsCount(Alert.GetValue()) + 1) * 4)"
notifications:
  - slack_default  # Set the webhook in /etc/crowdsec/notifications/slack.yaml before enabling this.
on_success: break
---

5: Notification template to show raw lines

format: |
  {{range . -}}
    {{range GetMeta . "raw" -}}
      {{.}}
    {{end -}}
  {{end -}}

The above template will iterate over each alert and key to dump them into the notification body. Other options I would suggest using is the group_threshold: 10 since if you do not it will spam a notification every single alert.

Drawbacks to using CrowdSec for not the intended purpose:

cscli alerts list without filters will become useless as it will show all alerts

+------+-------+---------------------+---------+----+-----------+-----------------------------------------+
|  ID  | value |       reason        | country | as | decisions |               created_at                |
+------+-------+---------------------+---------+----+-----------+-----------------------------------------+
| 1861 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.509423252 +0000 UTC |
| 1860 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.509311235 +0000 UTC |
| 1859 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.509054715 +0000 UTC |
| 1858 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.508737757 +0000 UTC |
| 1857 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.508581164 +0000 UTC |
| 1856 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.507880999 +0000 UTC |
| 1855 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.507516435 +0000 UTC |
| 1854 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.507295145 +0000 UTC |
| 1853 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.507184672 +0000 UTC |
| 1852 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.501442395 +0000 UTC |
| 1851 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.501318035 +0000 UTC |
| 1850 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.501172189 +0000 UTC |
| 1849 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.501098466 +0000 UTC |
| 1848 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.500983435 +0000 UTC |
| 1847 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.500783466 +0000 UTC |
| 1846 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.500675253 +0000 UTC |
| 1845 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.50054052 +0000 UTC  |
| 1844 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.500145738 +0000 UTC |
| 1843 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.499992259 +0000 UTC |
| 1842 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.499939758 +0000 UTC |
| 1841 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.499788858 +0000 UTC |
| 1840 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.499482766 +0000 UTC |
| 1839 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.499341501 +0000 UTC |
| 1838 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.499184742 +0000 UTC |
| 1837 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.499089219 +0000 UTC |
| 1836 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.498872092 +0000 UTC |
| 1835 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.498742034 +0000 UTC |
| 1834 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.498546261 +0000 UTC |
| 1833 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.498447402 +0000 UTC |
| 1832 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.498225645 +0000 UTC |
| 1831 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.498036177 +0000 UTC |
| 1830 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.497944237 +0000 UTC |
| 1829 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.497643397 +0000 UTC |
| 1828 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.497406333 +0000 UTC |
| 1827 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.497314436 +0000 UTC |
| 1826 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.497183044 +0000 UTC |
| 1825 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.496918471 +0000 UTC |
| 1824 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.496701316 +0000 UTC |
| 1823 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.496639739 +0000 UTC |
| 1822 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.496346228 +0000 UTC |
| 1821 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.496135023 +0000 UTC |
| 1820 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.496015699 +0000 UTC |
| 1819 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.495913378 +0000 UTC |
| 1818 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.495347778 +0000 UTC |
| 1817 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.495247694 +0000 UTC |
| 1816 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.495081549 +0000 UTC |
| 1815 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.494981388 +0000 UTC |
| 1814 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.49475985 +0000 UTC  |
| 1813 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.494648326 +0000 UTC |
| 1812 | error | me/journalctl-error |         |    |           | 2024-01-14 21:19:36.494476092 +0000 UTC |
+------+-------+---------------------+---------+----+-----------+-----------------------------------------+

My own advice would just make something together in bash scripting and cronjob it instead, all because you can do something with a software doesnt mean you should. However, let me know if you hit any errors.

3

Thank you very much for the detailed response. I will try it out.

My goal is to have a central log based IDS in a multi-server environment to replace the current OSSEC based one. But there are unknown security threats which have no ready made rules yet, software errors, hardware failure not recognized by monitoring tools or any other malfunction. That’s why I would like to do the following:

  1. Watch all the logs from all the units of all the servers. CrowdSec does this.
  2. Catch common security problems, do auto prevention. CrowdSec does this.
  3. The rest of the log lines, which were not catched in point 2, should be checked for words indicating malfunction. (failed, error, denied, refused and so on)
  4. Suppress known (no real problem) alerts from point 3. I think that’s what whitelists are for in CrowdSec.
  5. Notify on alerts. CrowdSec does this.

OSSEC does this by default via its level/priority based rule engine. For example if there is a log line from sshd containing “authentication failure” it processes that and the IPS part blocks the IP after many attempts (without notification if configure so). But if sshd logs “exotic never seen failure” I will be notified. This is a very handy feature to find new attacks, software errors and any malfunction. And using an another (even self-made) solution for that requires duplication of all the rules available in the IDS into the new solution to filter out those processed by the IDS. That’s the main reason why I want to integrate this into the IDS.

4

I managed to reach my goal. Your example was spot on. I only have one question left regarding the topic. Here is what I’ve done based on your kind help:

  1. Not labeled the lines at acquisition with onlyerrors. I let all the log lines into CrowdSec without labeling them for my use case.
  2. Made an s01 parser based on your example, but only filtering for my error words and not on the label.
  3. I named the parser file so, that it’s the last one in the folder sorted by filename (my filename starts with a ~)
  4. The rest is as you suggested.

Works like a charm. If any of the normal parsers grabs the event then it is sent to the next stage. But if not, then the last parser is mine, which checks for the bad words.

My question is that can I rely on the alphabetical order of the parser filenames? Or is there any configuration possibility to specify the parsers order?