Hi Everyone !
I’ve have a problem when I deploy crowdsec agent on Kubernetes with Helm.
failed to configure datasource file: could not create fsnotify watcher: too many open files"
I’ve problem systcl watcher user, Ok, so I’ve find this Problem config for acquisition
I’ve modified my systcl conf with this :
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-setup
namespace: kube-system
labels:
k8s-app: node-setup
spec:
selector:
matchLabels:
name: node-setup
template:
metadata:
labels:
name: node-setup
spec:
containers:
- name: node-setup
image: ubuntu
command: ["/bin/sh","-c"]
args: ["/script/node-setup.sh; while true; do echo Sleeping && sleep 3600; done"]
volumeMounts:
- name: node-setup-script
mountPath: /script
- name: dev
mountPath: /dev
- name: etc-lvm
mountPath: /etc/lvm
securityContext:
allowPrivilegeEscalation: true
privileged: true
volumes:
- name: node-setup-script
configMap:
name: node-setup-script
defaultMode: 0755
- name: dev
hostPath:
path: /dev
- name: etc-lvm
hostPath:
path: /etc/lvm
---
apiVersion: v1
kind: ConfigMap
metadata:
name: node-setup-script
namespace: kube-system
data:
node-setup.sh: |
#!/bin/bash
set -e
# change the file-watcher max-count on each node to 524288
# insert the new value into the system config
sysctl -w fs.inotify.max_user_watches=8192
# check that the new value was applied
cat /proc/sys/fs/inotify/max_user_watches
In agent pods or lapi if I make a cat of max user watches the values that modified. (old value: 128)
I’ve try destroy namespace, upgrade, I’ve all time same message
time="03-01-2024 21:35:04" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="03-01-2024 21:35:04" level=fatal msg="crowdsec init: while loading acquisition config: while configuring datasource of type file from /etc/crowdsec/acquis.yaml (position: 0): failed to configure datasource file: could not create fsnotify watcher: too many open files"
time="03-01-2024 21:35:04" level=fatal msg="crowdsec init: while loading acquisition config: while configuring datasource of type file from /etc/crowdsec/acquis.yaml (position: 0): failed to configure datasource file: could not create fsnotify watcher: too many open files"
Here my values yaml :
(I’ve tried set poll_without_inotify to true, but agent try it anyway with fsnotify)
container_runtime: containerd
agent:
# To specify each pod you want to process it logs (pods present in the node)
acquisition:
# The namespace where the pod is located
- namespace: ingress-nginx
# The pod name
podName: ingress-nginx-controller-*
# as in crowdsec configuration, we need to specify the program name so the parser will match and parse logs
program: nginx
poll_without_inotify: false
# resources:
# limits:
# memory: 100Mi
# requests:
# cpu: 150m
# memory: 100Mi
env:
- name: COLLECTIONS
value: "crowdsecurity/nginx"
lapi:
dashboard:
# -- Enable Metabase Dashboard (by default disabled)
enabled: true
image:
# -- docker image repository name
repository: metabase/metabase
# -- pullPolicy
pullPolicy: IfNotPresent
# -- docker image tag
tag: "v0.48.1"
# -- Metabase SQLite static DB containing Dashboards
assetURL: https://crowdsec-statics-assets.s3-eu-west-1.amazonaws.com/metabase_sqlite.zip
# -- Enable ingress object
ingress:
enabled: true
annotations:
# metabase only supports http so we need this annotation
nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
cert-manager.io/cluster-issuer: "letsencrypt-prod"
nginx.ingress.kubernetes.io/auth-secret: ingress-metabase-cs
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required"
# labels: {}
ingressClassName: "nginx" # nginx
host: "metabase.crowdsec.domain.fr"
tls:
- hosts:
- metabase.crowdsec.domain.fr
secretName: metabase-crowdsec-cert
# tls: {}
resources:
limits:
memory: 100Mi
requests:
cpu: 150m
memory: 100Mi
persistentVolume:
data:
enabled: true
accessModes:
- ReadWriteOnce
storageClassName: ""
existingClaim: ""
size: 10Gi
config:
enabled: true
accessModes:
- ReadWriteOnce
storageClassName: "scw-bssd"
existingClaim: "lapi-crowdsec-pvc"
size: 10Gi
Thank you !