Is my setup correct?

I wish you guys had videos explaining things. But below is what I have can you tell me if it looks right?

INFO[0000] Buckets Metrics:
+----------------------------+---------------+-----------+--------------+--------+---------+
|           BUCKET           | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+----------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/postfix-spam | -             | -         |           13 |     17 |      13 |
+----------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+--------------------------+------------+--------------+----------------+------------------------+
|          SOURCE          | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log        |       3853 | -            |           3853 | -                      |
| /var/log/mysql/error.log |         37 | -            |             37 | -                      |
| /var/log/syslog          |      43748 |           17 |          43731 |                     17 |
+--------------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+----------------------------------+-------+--------+----------+
|             PARSERS              | HITS  | PARSED | UNPARSED |
+----------------------------------+-------+--------+----------+
| child-crowdsecurity/postfix-logs |   322 |     17 |      305 |
| child-crowdsecurity/sshd-logs    |    85 | -      |       85 |
| crowdsecurity/dateparse-enrich   |    17 |     17 | -        |
| crowdsecurity/geoip-enrich       |    17 |     17 | -        |
| crowdsecurity/mysql-logs         |    37 | -      |       37 |
| crowdsecurity/non-syslog         |    37 |     37 | -        |
| crowdsecurity/postfix-logs       |   118 |     17 |      101 |
| crowdsecurity/sshd-logs          |    17 | -      |       17 |
| crowdsecurity/syslog-logs        | 47601 |  47601 | -        |
| crowdsecurity/whitelists         |    17 |     17 | -        |
+----------------------------------+-------+--------+----------+
INFO[0000] Local Api Metrics:
+----------------------+--------+-------+
|        ROUTE         | METHOD | HITS  |
+----------------------+--------+-------+
| /v1/alerts           | GET    |     7 |
| /v1/alerts/11        | GET    |     1 |
| /v1/alerts/12        | GET    |     1 |
| /v1/alerts/45        | GET    |     1 |
| /v1/decisions/stream | GET    | 24801 |
| /v1/watchers/login   | POST   |    34 |
+----------------------+--------+-------+
INFO[0000] Local Api Machines Metrics:
+----------------------------------+---------------+--------+------+
|             MACHINE              |     ROUTE     | METHOD | HITS |
+----------------------------------+---------------+--------+------+
| 74a9a1ba06e13820cbe5ac583b37c5c2 | /v1/alerts/11 | GET    |    1 |
| 74a9a1ba06e13820cbe5ac583b37c5c2 | /v1/alerts/12 | GET    |    1 |
| 74a9a1ba06e13820cbe5ac583b37c5c2 | /v1/alerts/45 | GET    |    1 |
| 74a9a1ba06e13820cbe5ac583b37c5c2 | /v1/alerts    | GET    |    7 |
+----------------------------------+---------------+--------+------+
INFO[0000] Local Api Bouncers Metrics:
+---------------------+----------------------+--------+-------+
|       BOUNCER       |        ROUTE         | METHOD | HITS  |
+---------------------+----------------------+--------+-------+
| cs-firewall-bouncer | /v1/decisions/stream | GET    | 24801 |
+---------------------+----------------------+--------+-------+


INFO[0000] Loaded 13 collecs, 18 parsers, 22 scenarios, 3 post-overflow parsers
INFO[0000] unmanaged items : 37 local, 0 tainted
INFO[0000] PARSERS:
**--------------------------------------------------------------------------------------------------------------**
** NAME                            📦 STATUS    VERSION  LOCAL PATH**
**--------------------------------------------------------------------------------------------------------------**
** crowdsecurity/modsecurity       ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/modsecurity.yaml**
** crowdsecurity/mysql-logs        ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/mysql-logs.yaml**
** crowdsecurity/whitelists        ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml**
** crowdsecurity/http-logs         ✔️  enabled  0.4      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml**
** crowdsecurity/iptables-logs     ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml**
** crowdsecurity/postfix-logs      ✔️  enabled  0.2      /etc/crowdsec/parsers/s01-parse/postfix-logs.yaml**
** crowdsecurity/sshd-logs         ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml**
** crowdsecurity/dateparse-enrich  ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml**
** crowdsecurity/postscreen-logs   ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/postscreen-logs.yaml**
** crowdsecurity/apache2-logs      ✔️  enabled  0.4      /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml**
** crowdsecurity/geoip-enrich      ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml**
** crowdsecurity/syslog-logs       ✔️  enabled  0.1      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml**
**--------------------------------------------------------------------------------------------------------------**
**INFO[0000] SCENARIOS:**
**---------------------------------------------------------------------------------------------------------------------------**
** NAME                                       📦 STATUS    VERSION  LOCAL PATH**
**---------------------------------------------------------------------------------------------------------------------------**
** ltsich/http-w00tw00t                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-w00tw00t.yaml**
** crowdsecurity/http-sqli-probing            ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml**
** crowdsecurity/http-backdoors-attempts      ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml**
** crowdsecurity/http-bad-user-agent          ✔️  enabled  0.3      /etc/crowdsec/scenarios/http-bad-user-agent.yaml**
** crowdsecurity/http-bf-wordpress_bf         ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-bf-wordpress_bf.yaml**
** crowdsecurity/http-crawl-non_statics       ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml**
** crowdsecurity/http-path-traversal-probing  ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml**
** crowdsecurity/http-xss-probing             ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml**
** crowdsecurity/modsecurity                  ✔️  enabled  0.2      /etc/crowdsec/scenarios/modsecurity.yaml**
** crowdsecurity/ssh-bf                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml**
** crowdsecurity/http-probing                 ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-probing.yaml**
** crowdsecurity/http-sensitive-files         ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml**
** crowdsecurity/iptables-scan-multi_ports    ✔️  enabled  0.1      /etc/crowdsec/scenarios/iptables-scan-multi_ports.yaml**
** crowdsecurity/mysql-bf                     ✔️  enabled  0.1      /etc/crowdsec/scenarios/mysql-bf.yaml**
** crowdsecurity/postfix-spam                 ✔️  enabled  0.2      /etc/crowdsec/scenarios/postfix-spam.yaml**
**---------------------------------------------------------------------------------------------------------------------------**
**INFO[0000] COLLECTIONS:**
**-------------------------------------------------------------------------------------------------------------**
** NAME                               📦 STATUS    VERSION  LOCAL PATH**
**-------------------------------------------------------------------------------------------------------------**
** crowdsecurity/apache2              ✔️  enabled  0.1      /etc/crowdsec/collections/apache2.yaml**
** crowdsecurity/iptables             ✔️  enabled  0.1      /etc/crowdsec/collections/iptables.yaml**
** crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml**
** crowdsecurity/postfix              ✔️  enabled  0.2      /etc/crowdsec/collections/postfix.yaml**
** crowdsecurity/sshd                 ✔️  enabled  0.1      /etc/crowdsec/collections/sshd.yaml**
** crowdsecurity/wordpress            ✔️  enabled  0.1      /etc/crowdsec/collections/wordpress.yaml**
** crowdsecurity/base-http-scenarios  ✔️  enabled  0.3      /etc/crowdsec/collections/base-http-scenarios.yaml**
** crowdsecurity/modsecurity          ✔️  enabled  0.1      /etc/crowdsec/collections/modsecurity.yaml**
** crowdsecurity/mysql                ✔️  enabled  0.1      /etc/crowdsec/collections/mysql.yaml**
**-------------------------------------------------------------------------------------------------------------**
**INFO[0000] POSTOVERFLOWS:**
**-----------------------------------------------------------------------------------------------------------------**
** NAME                         📦 STATUS    VERSION  LOCAL PATH**
**-----------------------------------------------------------------------------------------------------------------**
** crowdsecurity/cdn-whitelist  ✔️  enabled  0.3      /etc/crowdsec/postoverflows/s01-whitelist/cdn-whitelist.yaml**
**-----------------------------------------------------------------------------------------------------------------**


**-------------------------------------------------------------------------------------------------------------**
** NAME                               📦 STATUS    VERSION  LOCAL PATH**
**-------------------------------------------------------------------------------------------------------------**
** crowdsecurity/mysql                ✔️  enabled  0.1      /etc/crowdsec/collections/mysql.yaml**
** crowdsecurity/sshd                 ✔️  enabled  0.1      /etc/crowdsec/collections/sshd.yaml**
** crowdsecurity/base-http-scenarios  ✔️  enabled  0.3      /etc/crowdsec/collections/base-http-scenarios.yaml**
** crowdsecurity/linux                ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml**
** crowdsecurity/modsecurity          ✔️  enabled  0.1      /etc/crowdsec/collections/modsecurity.yaml**
** crowdsecurity/postfix              ✔️  enabled  0.2      /etc/crowdsec/collections/postfix.yaml**
** crowdsecurity/wordpress            ✔️  enabled  0.1      /etc/crowdsec/collections/wordpress.yaml**
** crowdsecurity/apache2              ✔️  enabled  0.1      /etc/crowdsec/collections/apache2.yaml**
** crowdsecurity/iptables             ✔️  enabled  0.1      /etc/crowdsec/collections/iptables.yaml**

Hello @tegra !

Please let me know if the documentation helps : Command line - Crowdsec

Would you do me a favor and tell me about the things that are confusing so we can improve documentation ?

Yes, we hope to produce this kind of videos soon, it’s just not easy to produce in decent quality.
Any suggestions of how you would like this format ?

Cheers,

youtube videos. Based on what I have setup does it look like its working?

The stuff above look good ?

yes sorry if I was not clear enough :slight_smile: your setup looks functional.

can you post an updated version ?

what matters is that :

  • the logs read from the files are not all unparsed. it’s normal to have some (or sometimes the majority) that is unparsed : crowdsec only parses the logs that are relevant to detect attacks
  • the scenarios are being at least instantiated and sometime overflow : it mean that you receive events eligible for the attack you want to detect

Here you go. It would be nice to get email reports.

INFO[0000] Buckets Metrics:
+----------------------------+---------------+-----------+--------------+--------+---------+
|           BUCKET           | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+----------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/postfix-spam | -             | -         |           28 |     37 |      28 |
+----------------------------+---------------+-----------+--------------+--------+---------+
INFO[0000] Acquisition Metrics:
+--------------------------+------------+--------------+----------------+------------------------+
|          SOURCE          | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log        |       8238 | -            |           8238 | -                      |
| /var/log/kern.log        |        157 | -            |            157 | -                      |
| /var/log/mysql/error.log |         37 | -            |             37 | -                      |
| /var/log/syslog          |      91003 |           37 |          90966 |                     37 |
+--------------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+----------------------------------+-------+--------+----------+
|             PARSERS              | HITS  | PARSED | UNPARSED |
+----------------------------------+-------+--------+----------+
| child-crowdsecurity/postfix-logs |   600 |     37 |      563 |
| child-crowdsecurity/sshd-logs    |   160 | -      |      160 |
| crowdsecurity/dateparse-enrich   |    37 |     37 | -        |
| crowdsecurity/geoip-enrich       |    35 |     35 | -        |
| crowdsecurity/iptables-logs      |   314 | -      |      314 |
| crowdsecurity/mysql-logs         |    37 | -      |       37 |
| crowdsecurity/non-syslog         |    37 |     37 | -        |
| crowdsecurity/postfix-logs       |   220 |     37 |      183 |
| crowdsecurity/sshd-logs          |    32 | -      |       32 |
| crowdsecurity/syslog-logs        | 99398 |  99398 | -        |
| crowdsecurity/whitelists         |    37 |     37 | -        |
+----------------------------------+-------+--------+----------+
INFO[0000] Local Api Metrics:
+----------------------+--------+-------+
|        ROUTE         | METHOD | HITS  |
+----------------------+--------+-------+
| /v1/alerts           | GET    |     8 |
| /v1/alerts/11        | GET    |     1 |
| /v1/alerts/12        | GET    |     1 |
| /v1/alerts/45        | GET    |     1 |
| /v1/decisions/stream | GET    | 50688 |
| /v1/watchers/login   | POST   |    35 |
+----------------------+--------+-------+
INFO[0000] Local Api Machines Metrics:
+----------------------------------+---------------+--------+------+
|             MACHINE              |     ROUTE     | METHOD | HITS |
+----------------------------------+---------------+--------+------+
| 74a9a17ba06e13820cbe5ac583b37c5a | /v1/alerts/12 | GET    |    1 |
| 74a9a17ba06e13820cbe5ac583b37c5a | /v1/alerts/45 | GET    |    1 |
| 74a9a17ba06e13820cbe5ac583b37c5a | /v1/alerts    | GET    |    8 |
| 74a9a17ba06e13820cbe5ac583b37c5a | /v1/alerts/11 | GET    |    1 |
+----------------------------------+---------------+--------+------+
INFO[0000] Local Api Bouncers Metrics:
+---------------------+----------------------+--------+-------+
|       BOUNCER       |        ROUTE         | METHOD | HITS  |
+---------------------+----------------------+--------+-------+
| cs-firewall-bouncer | /v1/decisions/stream | GET    | 50688 |
+---------------------+----------------------+--------+-------+

For email report you need to write your own bouncer.
What do you need ? Something like daily report ? hourly ?
If you send email on each ban this can be very verbose depending on the activity on your server.

Hopefully what I posted means it’s ok. I didn’t get someone to confirm that part.

A daily report would be good.

I can try to write a small bouncer in php who can do that.
if you can run php-cli it will work just fine.

If your server is sending mail without issue you just need a script that will get the new change on the CAPI and echo that. If it’s run through CRON you will get the output by mail.

You need to run something like that :
curl -X GET -H "X-Api-Key: ‘bouncer api key’’ -H “accept: application/json” “http://127.0.0.1:8080/v1/decisions/stream?startup=false

and manage to sort the output correctly.

Can you explain how will I do this. My linux is limited.

well this is more about scripting / programing that Linux…
You can query the LAPI from any device as long as you can join the LAPI.

To make it simple you need to make a GET request on the LAPI to get the last events.
You will get a JSON response that you have to parse in a “human readable form”.

[EDIT]
This is a very basic php script to give you the new blocked IP…
Simply run it through a CRON and if your mails system is ok you should receive the output by mail.
But I don’t know exactly how that will work, as LAPI will give you the new blocked IP since the last request, with a ban time of 4H by default, I don’t know if this will give you all new blocked IP even if they are expired or not… @thibault any idea about that ?

And this script won’t give you the ban time, only the IP and the scenario.

To work you need to add a bouncer : cscli bouncers add cs_mailer
This will give you the api-key that you have to change in the script

You need php-cli and php-curl.
run the script with php your script name.php

<?php

$headers = array(
            "X-Api-Key: xxxxxxxx",
            "accept: application/json");

/*Initialisation de la ressource curl*/
$c = curl_init();
/*On indique à curl quelle url on souhaite télécharger*/
curl_setopt($c, CURLOPT_URL, "http://127.0.0.1:8080/v1/decisions/stream?startup=false");
/*On indique à curl de nous retourner le contenu de la requête plutôt que de l'afficher*/
curl_setopt($c, CURLOPT_RETURNTRANSFER, true);
/*On indique à curl de ne pas retourner les headers http de la réponse dans la chaine de retour*/
curl_setopt($c, CURLOPT_HEADER, false);
/*On ignore les erreurs de certificat*/
curl_setopt($c, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($c, CURLOPT_SSL_VERIFYPEER, false);
// Set the content type to application/json
curl_setopt($c, CURLOPT_HTTPHEADER, $headers);	
/*On execute la requete*/
$output = curl_exec($c);
/*On a une erreur alors on la lève*/
if($output === false)
{
	trigger_error('Erreur curl : '.curl_error($c),E_USER_WARNING);
}
/*Si tout c'est bien passé on affiche le contenu de la requête*/
else
{
	//var_dump($output);
	$search = array(' ', "\t", "\n", "\r");
	$output = str_replace($search, '', $output);
	//echo $output . "\n";
	
	$output = json_decode($output,true);
	//print_r($output);

	//echo $output['deleted'][0]['value'] . "\n";

	$n = 0;
	$m = 0;

	$del = count($output['deleted']);
	//echo "deleted : $del \n";
	/*while($n < $del) {
		echo $output['deleted'][$n]['value'] . "\n";		
		$n = $n + 1;
	}*/

	$add = count($output['new']);
	echo "new : $add \n";
	

	while($m < $add) {
		echo $output['new'][$m]['value'] . "\t" . $output['new'][$m]['scenario'] . "\n";

		$m = $m + 1;
	}
}
	
	
	
?>

[/EDIT]