Too many open files on http-plugin

crowdsec
1

Hi guys,

I have the following errors on my logs

time="02-03-2022 12:32:11" level=error msg="Failed to make HTTP request : Post \"https://10.194.17.2:8080\": dial tcp 10.194.17.2:8080: socket: too many open files" @module=http-plugin
time="02-03-2022 12:32:11" level=error msg="rpc error: code = Unknown desc = Post \"https://10.194.17.2:8080\": dial tcp 10.194.17.2:8080: socket: too many open files error, retry num 1" plugin=http_elksiem
time="02-03-2022 12:32:12" level=error msg="rpc error: code = Unknown desc = Post \"https://10.194.17.2:8080\": dial tcp 10.194.17.2:8080: socket: too many open files" plugin:=http_elksiem

Here is my http.yaml notification file :

type: http

name: http_elksiem # this must match with the registered plugin in the profile
# Options include: trace, debug, info, warn, error, off
log_level: debug

format: |
  {{.|toJson}}
url: https://10.194.17.2:8080 # plugin will make requests to this url. Eg value https://www.example.com/

method: POST # eg either of "POST", "GET", "PUT" and other http verbs is valid value.

headers:
  Content-Type: "application/json"
  Authorization: "Basic MyLovelyPasswordYouWontKnow:)="

skip_tls_verification: true # either true or false. Default is false

timeout: 60s

This seems to happen after every alerts/notifications.
I use Crowdsec 1.3.0 on Ubuntu 20.04.3 LTS

Restarting Crowdsec temporary solve the issue…

Juste another little thing, at startup

time="02-03-2022 12:42:21" level=warning msg="Starting processing data"

Should be “info” level, not “warning” ?

2

I had the good idea to take a “lsof > beforerestart” and a “lsof > after”

I can see that :

[...]
notificat 1405123 1405217 notificat          nobody 1020u     IPv4         1467570638       0t0        TCP mysecretservername.europe-west1-b.c.mysecretprojectrname.internal:51484->10.194.17.2:http-alt (ESTABLISHED)
notificat 1405123 1405217 notificat          nobody 1021u     IPv4         1467576642       0t0        TCP mysecretservername.europe-west1-b.c.mysecretprojectrname.internal:56778->10.194.17.2:http-alt (ESTABLISHED)
notificat 1405123 1405217 notificat          nobody 1022u     IPv4         1467577348       0t0        TCP mysecretservername.europe-west1-b.c.mysecretprojectrname.internal:57462->10.194.17.2:http-alt (ESTABLISHED)
notificat 1405123 1405217 notificat          nobody 1023u     IPv4         1467580882       0t0        TCP mysecretservername.europe-west1-b.c.mysecretprojectrname.internal:60860->10.194.17.2:http-alt (ESTABLISHED)

Each time, it reaches the limit number of 1024 …
I can find 8192 lines like this (so 4*1024…)

So it seems it is making a new connection at each new notifications, and not closing the connections ?

3

Hello @Foxinou35 ,

To what server are you posting your payload ? From the http-notifier code, it seems that the connection is correctly closed :slight_smile:

4

Hi ! sorry for the late reply, I got no notification about your reply :-/

The remote server is a Logstash server (and then Elasticsearch after)

edit : When I do a netstat, I see 2 ESTABLISHED connexion to the logstash server
But If I do a lsof, I see 14 lines with ESTABLISHED.

edit2 : netstat : 3 ESTABLISHED, and 21 lines in lsof

So connexion seems to stay ESTABLISHED and there is problem with files not purged ?

5

Yes it is likely to be the source of the issue.
Do you know if logstash answers with a keepalive or equivalent that might force the go client to stay connected ?

6

I got a feedback from the guy who manages the Logstash, he says : did not find any parameter to control this behavior…

So cannot help :confused:

Do you have any idea ? or do you want me to do something to help you debug ?

Thanks

7

@Foxinou35 Could you please set log level of crowdsec from /etc/crowdsec/config.level to “debug” and paste the logs when this error happens ?

Edit: Typo fixed as pointed by @stratege1401

8

"debug" is better !