Reverse proxy duplicated access log

crowdsec
1

Hi,

we have an nginx that acts as a reverse proxy in front of apache2. I’ve configured apache so that it will log the X-Forwaded-For header that I set in nginx. So I have effectively duplicated access logs (one nginx, one apache).

I’ve configured crowdsec to read logs from both. So my question is how will crowdsec count events. Does it count events per log file or as a total. So if a request gets logged in nginx and apache will it effectively cut the threshold in half for decisions like: http-crawl-non_statics.

All the Best

Sebastian

PS: Thank you for this great tool.

For reference here is my acquis.yaml:

# nginx
filenames:
  - /var/log/nginx/access.log
  - /var/log/nginx/error.log
labels:
  type: nginx
---
# sshd
filenames:
  - /var/log/secure
labels:
  type: syslog
---
# syslog
filenames:
  - /var/log/messages
labels:
  type: syslog
---
# apache access log
filenames:
  - /opt/workdir/prod/logs/httpd/access_log
  - /opt/workdir/prod/logs/httpd/error_log
labels:
  type: apache2
2

For when it comes to scenarios since both are HTTP based it might count duplicated events. There a might since it also depends if you configured each logs file to log the FQDN if so the buckets are split into FQDN’s.

The question is why do you need to monitor both logs since they will basically log the same information.

3

Hi @iiAmLoz,

thanks for your reply. I’m currently reading from both logs because I was unsure how the error logs are parsed. I definitively need the nginx log (because it will contain requests limits) but the apache error logs will contain php-fpm errors and other stuff which I’m unsure if it is used by crowdsec.

But if this is of no use anyway then yeah I think just reading the nginx logs is the most logical approach. I could also read the apache error log without the apace access log if that helps.

All the Best

Sebastian