Is my setup correct?

tegra · February 8, 2021, 6:24pm

I wish you guys had videos explaining things. But below is what I have can you tell me if it looks right?

INFO[0000] Buckets Metrics:
+----------------------------+---------------+-----------+--------------+--------+---------+
|           BUCKET           | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+----------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/postfix-spam | -             | -         |           13 |     17 |      13 |
+----------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+--------------------------+------------+--------------+----------------+------------------------+
|          SOURCE          | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log        |       3853 | -            |           3853 | -                      |
| /var/log/mysql/error.log |         37 | -            |             37 | -                      |
| /var/log/syslog          |      43748 |           17 |          43731 |                     17 |
+--------------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+----------------------------------+-------+--------+----------+
|             PARSERS              | HITS  | PARSED | UNPARSED |
+----------------------------------+-------+--------+----------+
| child-crowdsecurity/postfix-logs |   322 |     17 |      305 |
| child-crowdsecurity/sshd-logs    |    85 | -      |       85 |
| crowdsecurity/dateparse-enrich   |    17 |     17 | -        |
| crowdsecurity/geoip-enrich       |    17 |     17 | -        |
| crowdsecurity/mysql-logs         |    37 | -      |       37 |
| crowdsecurity/non-syslog         |    37 |     37 | -        |
| crowdsecurity/postfix-logs       |   118 |     17 |      101 |
| crowdsecurity/sshd-logs          |    17 | -      |       17 |
| crowdsecurity/syslog-logs        | 47601 |  47601 | -        |
| crowdsecurity/whitelists         |    17 |     17 | -        |
+----------------------------------+-------+--------+----------+
INFO[0000] Local Api Metrics:
+----------------------+--------+-------+
|        ROUTE         | METHOD | HITS  |
+----------------------+--------+-------+
| /v1/alerts           | GET    |     7 |
| /v1/alerts/11        | GET    |     1 |
| /v1/alerts/12        | GET    |     1 |
| /v1/alerts/45        | GET    |     1 |
| /v1/decisions/stream | GET    | 24801 |
| /v1/watchers/login   | POST   |    34 |
+----------------------+--------+-------+
INFO[0000] Local Api Machines Metrics:
+----------------------------------+---------------+--------+------+
|             MACHINE              |     ROUTE     | METHOD | HITS |
+----------------------------------+---------------+--------+------+
| 74a9a1ba06e13820cbe5ac583b37c5c2 | /v1/alerts/11 | GET    |    1 |
| 74a9a1ba06e13820cbe5ac583b37c5c2 | /v1/alerts/12 | GET    |    1 |
| 74a9a1ba06e13820cbe5ac583b37c5c2 | /v1/alerts/45 | GET    |    1 |
| 74a9a1ba06e13820cbe5ac583b37c5c2 | /v1/alerts    | GET    |    7 |
+----------------------------------+---------------+--------+------+
INFO[0000] Local Api Bouncers Metrics:
+---------------------+----------------------+--------+-------+
|       BOUNCER       |        ROUTE         | METHOD | HITS  |
+---------------------+----------------------+--------+-------+
| cs-firewall-bouncer | /v1/decisions/stream | GET    | 24801 |
+---------------------+----------------------+--------+-------+


INFO[0000] Loaded 13 collecs, 18 parsers, 22 scenarios, 3 post-overflow parsers
INFO[0000] unmanaged items : 37 local, 0 tainted
INFO[0000] PARSERS:
**--------------------------------------------------------------------------------------------------------------**
** NAME                            📦 STATUS    VERSION  LOCAL PATH**
**--------------------------------------------------------------------------------------------------------------**
** crowdsecurity/modsecurity       ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/modsecurity.yaml**
** crowdsecurity/mysql-logs        ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/mysql-logs.yaml**
** crowdsecurity/whitelists        ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml**
** crowdsecurity/http-logs         ✔️  enabled  0.4      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml**
** crowdsecurity/iptables-logs     ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml**
** crowdsecurity/postfix-logs      ✔️  enabled  0.2      /etc/crowdsec/parsers/s01-parse/postfix-logs.yaml**
** crowdsecurity/sshd-logs         ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml**
** crowdsecurity/dateparse-enrich  ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml**
** crowdsecurity/postscreen-logs   ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/postscreen-logs.yaml**
** crowdsecurity/apache2-logs      ✔️  enabled  0.4      /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml**
** crowdsecurity/geoip-enrich      ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml**
** crowdsecurity/syslog-logs       ✔️  enabled  0.1      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml**
**--------------------------------------------------------------------------------------------------------------**
**INFO[0000] SCENARIOS:**
**---------------------------------------------------------------------------------------------------------------------------**
** NAME                                       📦 STATUS    VERSION  LOCAL PATH**
**---------------------------------------------------------------------------------------------------------------------------**
** ltsich/http-w00tw00t                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-w00tw00t.yaml**
** crowdsecurity/http-sqli-probing            ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml**
** crowdsecurity/http-backdoors-attempts      ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml**
** crowdsecurity/http-bad-user-agent          ✔️  enabled  0.3      /etc/crowdsec/scenarios/http-bad-user-agent.yaml**
** crowdsecurity/http-bf-wordpress_bf         ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-bf-wordpress_bf.yaml**
** crowdsecurity/http-crawl-non_statics       ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml**
** crowdsecurity/http-path-traversal-probing  ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml**
** crowdsecurity/http-xss-probing             ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml**
** crowdsecurity/modsecurity                  ✔️  enabled  0.2      /etc/crowdsec/scenarios/modsecurity.yaml**
** crowdsecurity/ssh-bf                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml**
** crowdsecurity/http-probing                 ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-probing.yaml**
** crowdsecurity/http-sensitive-files         ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml**
** crowdsecurity/iptables-scan-multi_ports    ✔️  enabled  0.1      /etc/crowdsec/scenarios/iptables-scan-multi_ports.yaml**
** crowdsecurity/mysql-bf                     ✔️  enabled  0.1      /etc/crowdsec/scenarios/mysql-bf.yaml**
** crowdsecurity/postfix-spam                 ✔️  enabled  0.2      /etc/crowdsec/scenarios/postfix-spam.yaml**
**---------------------------------------------------------------------------------------------------------------------------**
**INFO[0000] COLLECTIONS:**
**-------------------------------------------------------------------------------------------------------------**
** NAME                               📦 STATUS    VERSION  LOCAL PATH**
**-------------------------------------------------------------------------------------------------------------**
** crowdsecurity/apache2              ✔️  enabled  0.1      /etc/crowdsec/collections/apache2.yaml**
** crowdsecurity/iptables             ✔️  enabled  0.1      /etc/crowdsec/collections/iptables.yaml**
** crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml**
** crowdsecurity/postfix              ✔️  enabled  0.2      /etc/crowdsec/collections/postfix.yaml**
** crowdsecurity/sshd                 ✔️  enabled  0.1      /etc/crowdsec/collections/sshd.yaml**
** crowdsecurity/wordpress            ✔️  enabled  0.1      /etc/crowdsec/collections/wordpress.yaml**
** crowdsecurity/base-http-scenarios  ✔️  enabled  0.3      /etc/crowdsec/collections/base-http-scenarios.yaml**
** crowdsecurity/modsecurity          ✔️  enabled  0.1      /etc/crowdsec/collections/modsecurity.yaml**
** crowdsecurity/mysql                ✔️  enabled  0.1      /etc/crowdsec/collections/mysql.yaml**
**-------------------------------------------------------------------------------------------------------------**
**INFO[0000] POSTOVERFLOWS:**
**-----------------------------------------------------------------------------------------------------------------**
** NAME                         📦 STATUS    VERSION  LOCAL PATH**
**-----------------------------------------------------------------------------------------------------------------**
** crowdsecurity/cdn-whitelist  ✔️  enabled  0.3      /etc/crowdsec/postoverflows/s01-whitelist/cdn-whitelist.yaml**
**-----------------------------------------------------------------------------------------------------------------**


**-------------------------------------------------------------------------------------------------------------**
** NAME                               📦 STATUS    VERSION  LOCAL PATH**
**-------------------------------------------------------------------------------------------------------------**
** crowdsecurity/mysql                ✔️  enabled  0.1      /etc/crowdsec/collections/mysql.yaml**
** crowdsecurity/sshd                 ✔️  enabled  0.1      /etc/crowdsec/collections/sshd.yaml**
** crowdsecurity/base-http-scenarios  ✔️  enabled  0.3      /etc/crowdsec/collections/base-http-scenarios.yaml**
** crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml**
** crowdsecurity/modsecurity          ✔️  enabled  0.1      /etc/crowdsec/collections/modsecurity.yaml**
** crowdsecurity/postfix              ✔️  enabled  0.2      /etc/crowdsec/collections/postfix.yaml**
** crowdsecurity/wordpress            ✔️  enabled  0.1      /etc/crowdsec/collections/wordpress.yaml**
** crowdsecurity/apache2              ✔️  enabled  0.1      /etc/crowdsec/collections/apache2.yaml**
** crowdsecurity/iptables             ✔️  enabled  0.1      /etc/crowdsec/collections/iptables.yaml**

thibault · February 9, 2021, 7:59am

Hello @tegra !

Please let me know if the documentation helps : CrowdSec | CrowdSec

Would you do me a favor and tell me about the things that are confusing so we can improve documentation ?

Yes, we hope to produce this kind of videos soon, it’s just not easy to produce in decent quality.
Any suggestions of how you would like this format ?

Cheers,

tegra · February 9, 2021, 1:35pm

youtube videos. Based on what I have setup does it look like its working?

tegra · February 10, 2021, 11:26pm

The stuff above look good ?

thibault · February 11, 2021, 8:31am

yes sorry if I was not clear enough your setup looks functional.

can you post an updated version ?

what matters is that :

the logs read from the files are not all unparsed. it’s normal to have some (or sometimes the majority) that is unparsed : crowdsec only parses the logs that are relevant to detect attacks
the scenarios are being at least instantiated and sometime overflow : it mean that you receive events eligible for the attack you want to detect

tegra · February 11, 2021, 6:15pm

Here you go. It would be nice to get email reports.

INFO[0000] Buckets Metrics:
+----------------------------+---------------+-----------+--------------+--------+---------+
|           BUCKET           | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+----------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/postfix-spam | -             | -         |           28 |     37 |      28 |
+----------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+--------------------------+------------+--------------+----------------+------------------------+
|          SOURCE          | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log        |       8238 | -            |           8238 | -                      |
| /var/log/kern.log        |        157 | -            |            157 | -                      |
| /var/log/mysql/error.log |         37 | -            |             37 | -                      |
| /var/log/syslog          |      91003 |           37 |          90966 |                     37 |
+--------------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+----------------------------------+-------+--------+----------+
|             PARSERS              | HITS  | PARSED | UNPARSED |
+----------------------------------+-------+--------+----------+
| child-crowdsecurity/postfix-logs |   600 |     37 |      563 |
| child-crowdsecurity/sshd-logs    |   160 | -      |      160 |
| crowdsecurity/dateparse-enrich   |    37 |     37 | -        |
| crowdsecurity/geoip-enrich       |    35 |     35 | -        |
| crowdsecurity/iptables-logs      |   314 | -      |      314 |
| crowdsecurity/mysql-logs         |    37 | -      |       37 |
| crowdsecurity/non-syslog         |    37 |     37 | -        |
| crowdsecurity/postfix-logs       |   220 |     37 |      183 |
| crowdsecurity/sshd-logs          |    32 | -      |       32 |
| crowdsecurity/syslog-logs        | 99398 |  99398 | -        |
| crowdsecurity/whitelists         |    37 |     37 | -        |
+----------------------------------+-------+--------+----------+
INFO[0000] Local Api Metrics:
+----------------------+--------+-------+
|        ROUTE         | METHOD | HITS  |
+----------------------+--------+-------+
| /v1/alerts           | GET    |     8 |
| /v1/alerts/11        | GET    |     1 |
| /v1/alerts/12        | GET    |     1 |
| /v1/alerts/45        | GET    |     1 |
| /v1/decisions/stream | GET    | 50688 |
| /v1/watchers/login   | POST   |    35 |
+----------------------+--------+-------+
INFO[0000] Local Api Machines Metrics:
+----------------------------------+---------------+--------+------+
|             MACHINE              |     ROUTE     | METHOD | HITS |
+----------------------------------+---------------+--------+------+
| 74a9a17ba06e13820cbe5ac583b37c5a | /v1/alerts/12 | GET    |    1 |
| 74a9a17ba06e13820cbe5ac583b37c5a | /v1/alerts/45 | GET    |    1 |
| 74a9a17ba06e13820cbe5ac583b37c5a | /v1/alerts    | GET    |    8 |
| 74a9a17ba06e13820cbe5ac583b37c5a | /v1/alerts/11 | GET    |    1 |
+----------------------------------+---------------+--------+------+
INFO[0000] Local Api Bouncers Metrics:
+---------------------+----------------------+--------+-------+
|       BOUNCER       |        ROUTE         | METHOD | HITS  |
+---------------------+----------------------+--------+-------+
| cs-firewall-bouncer | /v1/decisions/stream | GET    | 50688 |
+---------------------+----------------------+--------+-------+

Sich · February 18, 2021, 1:21pm

For email report you need to write your own bouncer.
What do you need ? Something like daily report ? hourly ?
If you send email on each ban this can be very verbose depending on the activity on your server.

tegra · February 18, 2021, 2:29pm

Hopefully what I posted means it’s ok. I didn’t get someone to confirm that part.

A daily report would be good.

Sich · February 18, 2021, 2:57pm

I can try to write a small bouncer in php who can do that.
if you can run php-cli it will work just fine.

If your server is sending mail without issue you just need a script that will get the new change on the CAPI and echo that. If it’s run through CRON you will get the output by mail.

You need to run something like that :
curl -X GET -H "X-Api-Key: ‘bouncer api key’’ -H “accept: application/json” “http://127.0.0.1:8080/v1/decisions/stream?startup=false”

and manage to sort the output correctly.

tegra · February 18, 2021, 4:53pm

Can you explain how will I do this. My linux is limited.

Sich · February 18, 2021, 7:21pm

well this is more about scripting / programing that Linux…
You can query the LAPI from any device as long as you can join the LAPI.

To make it simple you need to make a GET request on the LAPI to get the last events.
You will get a JSON response that you have to parse in a “human readable form”.

[EDIT]
This is a very basic php script to give you the new blocked IP…
Simply run it through a CRON and if your mails system is ok you should receive the output by mail.
But I don’t know exactly how that will work, as LAPI will give you the new blocked IP since the last request, with a ban time of 4H by default, I don’t know if this will give you all new blocked IP even if they are expired or not… @thibault any idea about that ?

And this script won’t give you the ban time, only the IP and the scenario.

To work you need to add a bouncer : cscli bouncers add cs_mailer
This will give you the api-key that you have to change in the script

You need php-cli and php-curl.
run the script with php your script name.php

<?php

$headers = array(
            "X-Api-Key: xxxxxxxx",
            "accept: application/json");

/*Initialisation de la ressource curl*/
$c = curl_init();
/*On indique à curl quelle url on souhaite télécharger*/
curl_setopt($c, CURLOPT_URL, "http://127.0.0.1:8080/v1/decisions/stream?startup=false");
/*On indique à curl de nous retourner le contenu de la requête plutôt que de l'afficher*/
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
/*On indique à curl de ne pas retourner les headers http de la réponse dans la chaine de retour*/
curl_setopt($c, CURLOPT_HEADER, false);
/*On ignore les erreurs de certificat*/
curl_setopt($c, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($c, CURLOPT_SSL_VERIFYPEER, false);
// Set the content type to application/json
curl_setopt($c, CURLOPT_HTTPHEADER, $headers);	
/*On execute la requete*/
$output = curl_exec($c);
/*On a une erreur alors on la lève*/
if($output === false)
{
	trigger_error('Erreur curl : '.curl_error($c),E_USER_WARNING);
}
/*Si tout c'est bien passé on affiche le contenu de la requête*/
else
{
	//var_dump($output);
	$search = array(' ', "\t", "\n", "\r");
	$output = str_replace($search, '', $output);
	//echo $output . "\n";
	
	$output = json_decode($output,true);
	//print_r($output);

	//echo $output['deleted'][0]['value'] . "\n";

	$n = 0;
	$m = 0;

	$del = count($output['deleted']);
	//echo "deleted : $del \n";
	/*while($n < $del) {
		echo $output['deleted'][$n]['value'] . "\n";		
		$n = $n + 1;
	}*/

	$add = count($output['new']);
	echo "new : $add \n";
	

	while($m < $add) {
		echo $output['new'][$m]['value'] . "\t" . $output['new'][$m]['scenario'] . "\n";

		$m = $m + 1;
	}
}
	
	
	
?>

[/EDIT]

Topic		Replies	Views
Question about how to read the metrics	11	2765	January 27, 2021
New install, postfix / IMAP	22	4386	March 4, 2021
Keycloak collection is not banning crowdsec	1	257	December 20, 2023
Help: having only unparsed logs	7	3382	April 8, 2021
Local Api Metrics output has hundreds of lines! crowdsec	1	239	September 29, 2023

Is my setup correct?

Related Topics