Ipset, performance, whitelist, api, dashboard, postfix logs

[Sich]

Hi, lot of questions here

I use ipset (and nftables with firewalld), I use a small in house script with the custom bouncer.
All work fine. Do you plan to write some ipset bouncer ? (2 commands ipset add and ipset del).

About performance, I use fail2ban since… years and years…
Do you have some test to see how run crowdsec against fail2ban ?
Fail2ban is well know, ton of scripts, crowdsec is new, we don’t know if the project will survive. At least we need some information about how it work

About whitelist, I have made a copy of cdn-whitelist.yaml to use my own ip list.
But the file in the source_url was not downloaded, I had to do that manually.
Is there any automatic process on that ? Some check for a new version sometimes ?
Or do I have to manage that on my side ?

With the API.
In the doc I see : cscli api pull
If I run the command this download a list of IP.
But do I need to run that by myself (or with cron) ?
Or there is a default automatic check ?

Other command : cscli api enroll
What is the crowdsec backend ? Where can I register some user account ?

About the dashboard, we need to run docker… But I don’t want to run docker just for that (yeah I’m some old school guy), do you have some documentation to configure that by myself ?

And last point, when you install crowdsec and you select postfix filter, the log file is not configured in acquis.yaml, I had to make this myself (not a big deal but for new user…).

Sorry for all those questions !

[EDIT]
Sorry, last question !
How much IP adress (or subnet) can you whitelist ?
I have a big list of IP from Ezoic (more that 1200), is this a problem ?
Fail2ban didn’t like when I try to add all this list on the ignoreip…
[/EDIT]

[alteredCoder]

Hello @Sich

First thank you for using crowdsec and asking those questions.
I will answer your questions point by point, but i would like you to know that we are soon releasing a new version of crowdsec that will answer some of your problems

• Do you plan to write some ipset bouncer

The cs-netfilter-blocker use ipset , so i think you can use it

• Do you have some test to see how run crowdsec against fail2ban ?

We are going to release an article soon about crowdsec vs fail2ban against a botnet

• Is there any automatic process on that ? Some check for a new version sometimes ? Or do I have to manage that on my side ?

Indeed, currently the source_url is downloaded when you install a parser with cscli. About custom parsers, there is no way to download the source_url except manually. There is currently no easy process to upgrade the source file, thank you for pointing it out the issue. We are going to think about this problem internally to find a way to fix this.

• cscli api pull : But do I need to run that by myself (or with cron) ?

When installing crowdsec with the wizard.sh , a cron is deployed to run this command.
In the future release of crowdsec, this will be done automatically without cron.

• What is the crowdsec backend ? Where can I register some user account ?

Nice spot! Indeed we have the feature in cscli but the backend is now too minimalist to be online. We will communicate when this backend will be released, but i can’t confirm a date for now.

• About the dashboard, we need to run docker… But I don’t want to run docker just for that (yeah I’m some old school guy), do you have some documentation to configure that by myself ?

Indeed there is no documentation on how to configure this about yourself. If you plan to give it a shot, please let us know so we can assist you and document it for future users.

• And last point, when you install crowdsec and you select postfix filter, the log file is not configured in acquis.yaml, I had to make this myself (not a big deal but for new user…).

It is planned to improve the acquisition file(s) management so that it can be part of a collection. The goal is to avoid to force users to edit this file manually when you add support for new services, but i can’t give you an ETA right now.

• How much IP adress (or subnet) can you whitelist ? I have a big list of IP from Ezoic (more that 1200), is this a problem ?

To be honest, we didn’t try to whitelist as much IP addresses, but i think you can do it. To warn you, this may use a significant amount of RAM.

I hope i’ve answered your questions, and thank you again for your interest

[Sich]

@alteredCoder

The cs-netfilter-blocker use ipset , so i think you can use it

Hum, the bouncer use iptable. That’s not really a issue because iptable is use as an alias for nftables, but I want to stay on something 100% nftables atm. My script work fine, don’t worry

@alteredCoder

We are going to release an article soon about crowdsec vs fail2ban against a botnet

oh that’s a good news !

@alteredCoder

Indeed, currently the source_url is downloaded when you install a parser with cscli. About custom parsers, there is no way to download the source_url except manually. There is currently no easy process to upgrade the source file, thank you for pointing it out the issue. We are going to think about this problem internally to find a way to fix this.

I can manage that myself it’s really not a problem, I just have to know that I need to do it myself !

@alteredCoder

When installing crowdsec with the wizard.sh , a cron is deployed to run this command.
In the future release of crowdsec, this will be done automatically without cron.

Yep, I have found that cron.

@alteredCoder

Nice spot! Indeed we have the feature in cscli but the backend is now too minimalist to be online. We will communicate when this backend will be released, but i can’t confirm a date for now.

No problem, just to be sure that I didn’t miss something

@alteredCoder

Indeed there is no documentation on how to configure this about yourself. If you plan to give it a shot, please let us know so we can assist you and document it for future users.

If you can it can be usefull, but no rush, when you have time.
One thing just, I didn’t find any reference in the database (I use crowdsec with mysql) about the server who have initiate the ban. That can be usefull as I plan to use 1 central database for around 40 servers.
And if that can be displayed in the dashboard.

@alteredCoder

It is planned to improve the acquisition file(s) management so that it can be part of a collection. The goal is to avoid to force users to edit this file manually when you add support for new services, but i can’t give you an ETA right now.

It was just to let you know that the installer doesn’t add the postfix log files.

@alteredCoder

To be honest, we didn’t try to whitelist as much IP addresses, but i think you can do it. To warn you, this may use a significant amount of RAM.

I will test that… That huge IP address is a big issue for me actually (fail2ban don’t like that huge list at all) as client add website behind ezoic and doesn’t set the log properly, and then fail2ban ban the servers from ezoic… If at least I can whitelist those ips…

Thx for all your answers, your project is interesting, in the past I had build something similar (for my personnal use) with fail2ban. With crowdsec the central database make the management far more easier. And having a direct line with the dev is a very good point too.

[thibault]

@Sich

If you can it can be usefull, but no rush, when you have time.
One thing just, I didn’t find any reference in the database (I use crowdsec with mysql) about the server who have initiate the ban. That can be usefull as I plan to use 1 central database for around 40 servers.
And if that can be displayed in the dashboard.

This is very interesting, as one the main change of the next crowdsec release will be to expose a local REST api that can be consumed by bouncers : this will allow bouncers to be stateless and hopefully make the management of such setups a lot easier. Would you be interested into beta-testing this feature ?

[Sich]

Hum, my idea was that the CrowdSec service send is id when he add a ban on the database, to be able to know where the “attack” was.
It’s useless on a single host installation, but in my configuration this can be very usefull.

The bouncer don’t really need that information, it’s more for administrative task to understand what happen and where.

And about some api on the bouncer why not, atm I only use crowdsec on 1 testing server, with the database on another server, communication is made through a vpn.
But I can slowly switch on some small server in production to see how this will work in “real life”, because testing server can’t really reproduce how it’s going in production.