Problem config for acquisition

crowdsec
1

Hi,

My crowdsec container is not stable anymore…

I have this logs before it restart :

time="17-04-2023 16:35:37" level=trace msg="waiting for stdio data"

time="17-04-2023 16:35:38" level=fatal msg="crowdsec init: Error while loading acquisition config : while configuring datasource of type file from /etc/crowdsec/acquis.yaml (position: 0): failed to configure datasource file: Could not create fsnotify watcher: too many open files"

time="17-04-2023 16:35:38" level=fatal msg="crowdsec init: Error while loading acquisition config : while configuring datasource of type file from /etc/crowdsec/acquis.yaml (position: 0): failed to configure datasource file: Could not create fsnotify watcher: too many open files"

Regenerate local agent credentials

Do you have any idea that could help me resolve this ?

My acquis.yml :

filenames:
  - /var/log/auth.log
labels:
  type: syslog
---
filenames:
  - /var/log/traefik/*.log
labels:
  type: traefik
---
filenames:
  - /var/log/pure-ftpd/pureftpd.log
labels:
  type: pure-ftpd

I use the latest version of crowdsec docker image.

Everything was working fine until the server was restarted.

Thanks

2

It seems the linux process fs notify limit is reached. Do you have a lot of files within /var/log/traefik/? cause that is the only wildcard I can see. If not you can raise the fsnotify limit

3

Hi @iiAmLoz ,

Thanks to taking the time to answer me.

I have try to use the acquis.d folder like in the doc.
I have chunk the acquis.yaml file into 3 files pureftpd.yaml, ssh.yaml and traefik.yaml.

Same problem but now it pointed on this files.
BTW, i need to create and share an empty acquis.yaml to this to work.
This is not explain in the doc…

Thanks to your comment, I checked if I had the same problem by using only the ssh.yaml file.
And unfortunately, it is the case.

time="20-04-2023 09:08:36" level=fatal msg="crowdsec init: Error while loading acquisition config : while configuring datasource of type file from /etc/crowdsec/acquis.d/ssh.yaml (position: 0): failed to configure datasource file: Could not create fsnotify watcher: too many open files"

my ssh.yaml file

filenames:
  - /var/log/auth.log
  - /var/log/syslog
labels:
  type: syslog

I have a log rotate on this files (and also on traefik log file, and pureftpd log file).

-rw-r-----  1 root        adm              67K 20 avril 09:07 auth.log
-rw-r-----  1 root        adm              57K 17 avril 15:45 auth.log.1
-rw-r-----  1 root        adm             7,1K  8 avril 23:17 auth.log.2.gz
-rw-r-----  1 root        adm             6,8K  1 avril 23:17 auth.log.3.gz
-rw-r-----  1 root        adm             6,0K 25 mars  23:17 auth.log.4.gz

And

-rw-r-----  1 root        adm             5,8M 20 avril 09:09 syslog
-rw-r-----  1 root        adm             9,9M 17 avril 15:45 syslog.1
-rw-r-----  1 root        adm             2,1M  9 avril 00:00 syslog.2.gz
-rw-r-----  1 root        adm             427K  2 avril 00:00 syslog.3.gz
-rw-r-----  1 root        adm             436K 26 mars  00:00 syslog.4.gz
-rw-r-----  1 root        adm              951 11 nov.   2021 syslog.6.gz
-rw-r-----  1 root        adm              891 10 nov.   2021 syslog.7.gz

Same for traefik or pureftpd logs.

I dont know what to do :disappointed_relieved:

Cheers

4

@iiAmLoz ,

i have futher infos.

If i tail auth.log, i have this error:

failed to create fsnotify watcher: too many open files

I seems that i dont have enough watchers.

cat /proc/sys/fs/inotify/max_user_watches
8192

Maybe it’s because i use loki with promtail and there is too many files opens in the overall system ?

I will check if i stop my loki services.

I will give you an update.

5

That’s it !

I have to many services that open logs files… my monitoring system (grok and promtail).

It was not related to crowdsec.

There is what i do to resolve the issue:

I had to increase the inotify max_user_instances.

$ cat /proc/sys/fs/inotify/max_user_instances 
128
$ sudo sysctl fs.inotify.max_user_instances=8192
fs.inotify.max_user_instances = 8192
$ cat /proc/sys/fs/inotify/max_user_instances 
8192

If you want to do it permanently:

$ echo fs.inotify.max_user_instances=8192 >> /etc/sysctl.conf
$ sysctl --load

i don’t know exactly what number i have to use but it seems to work.

Bye