Hi ! I ran into a problem, I want the crowdsec to monitor nginx logs from a directory other than the standard directory. Trying to solve this problem I specified the my logs directory in wizard.sh “log_locations[nginx]=”. After installation in acquis.yaml, lines with my directory logs appeared, but cscli metrics tells me no nginx metrics found.Please, tell me what I’m doing wrong and how to connect the non-standard logs directory correctly.
Hello @Fleedstix 
You can directly indicate the logs you want to monitor in acquis.yaml.
Can you show cscli metrics acquisition part ? it should display how many lines are read from a given log files, how many are parsed etc.
Given this, we can guess if the logs are read but not parsed, or not read at all. If it’s the first case, it might be a log format related issue !
my cscli metrics:
in acquis.yaml :
#Generated acquisition file - wizard.sh (service: nginx) / files : /opt/myapp/logs/error.log /opt/myapp/logs/access.log
filenames:
- /opt/myapp/logs/error.log
- /opt/myapp/logs/access.log
labels:
type: nginx
—
Can you paste the acquis.yaml using blockquotes from the editor interface please ? (seems the file is messed up, but hard to tell as indentation is not preserved).
From your cscli metrics it seems that the files /opt/myapp/logs/error.log you mentionned are not read at all, like they don’t exist. Can you share crowdsec startup log as well ?
Thanks,
Look here please
GNU nano 4.8 acquis.yaml
> #Generated acquisition file - wizard.sh (service: nginx) / files : /opt/myapp/logs/error.log /opt/myapp/logs/access.log
> filenames:
> - /opt/myapp/logs/error.log
> - /opt/myapp/logs/access.log
> labels:
> type: nginx
> ---
> #Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/auth.log
> filenames:
> - /var/log/auth.log
> labels:
> type: syslog
> ---
> #Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log
> filenames:
> - /var/log/syslog
> - /var/log/kern.log
> labels:
> type: syslog
> ---after systemctl restart crowdsec:
time=“22-03-2021 16:55:32” level=warning msg=“Crowdsec service shutting down”
time=“22-03-2021 16:55:32” level=info msg=“Crowdsec v1.0.9-a8b16a66b110ebe03bb330cda2600226a3a862d7”
time=“22-03-2021 16:55:32” level=info msg=“Loading prometheus collectors”
time=“22-03-2021 16:55:32” level=info msg=“Loading CAPI pusher”
time=“22-03-2021 16:55:32” level=info msg=“Loading grok library /etc/crowdsec//patterns/”
time=“22-03-2021 16:55:33” level=info msg=“Loading enrich plugins”
time=“22-03-2021 16:55:33” level=info msg=“Loading parsers 7 stages”
time=“22-03-2021 16:55:33” level=info msg=“Node in /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml has no name,author or description. Skipping.”
time=“22-03-2021 16:55:33” level=info msg=“Loaded 2 parser nodes” file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
time=“22-03-2021 16:55:33” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s01-parse/nginx-logs.yaml
time=“22-03-2021 16:55:33” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
time=“22-03-2021 16:55:33” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
time=“22-03-2021 16:55:33” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
time=“22-03-2021 16:55:33” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s02-enrich/http-logs.yaml
time=“22-03-2021 16:55:33” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml
time=“22-03-2021 16:55:33” level=info msg=“Loaded 8 nodes, 3 stages”
time=“22-03-2021 16:55:33” level=info msg=“Loading postoverflow Parsers”
time=“22-03-2021 16:55:33” level=info msg=“Loaded 0 nodes, 0 stages”
time=“22-03-2021 16:55:33” level=info msg=“Loading 11 scenario files”
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=rough-sky file=/etc/crowdsec/scenarios/http-sensitive-files.yaml name=crowdsecurity/http-sensitive-files
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=restless-pine file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=falling-snow file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=dark-sun file=/etc/crowdsec/scenarios/http-path-traversal-probing.yaml name=crowdsecurity/http-path-traversal-probing
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=misty-voice file=/etc/crowdsec/scenarios/http-sqli-probing.yaml name=crowdsecurity/http-sqli-probbing-detection
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=broken-sun file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=withered-cherry file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=icy-firefly file=/etc/crowdsec/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=white-shape file=/etc/crowdsec/scenarios/http-crawl-non_statics.yaml name=crowdsecurity/http-crawl-non_statics
time=“22-03-2021 16:55:33” level=info msg=“Adding trigger bucket” cfg=green-pond file=/etc/crowdsec/scenarios/http-w00tw00t.yaml name=ltsich/http-w00tw00t
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=withered-surf file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time=“22-03-2021 16:55:33” level=info msg=“Adding leaky bucket” cfg=aged-silence file=/etc/crowdsec/scenarios/http-probing.yaml name=crowdsecurity/http-probing
time=“22-03-2021 16:55:33” level=warning msg=“Loaded 12 scenarios”
time=“22-03-2021 16:55:33” level=info msg=“loading acquisition file : /etc/crowdsec/acquis.yaml”
time=“22-03-2021 16:55:33” level=info msg=“[file datasource] opening file ‘/opt/myapp/logs/error.log’”
time=“22-03-2021 16:55:33” level=info msg=“[file datasource] opening file ‘/opt/myapp/logs/access.log’”
time=“22-03-2021 16:55:33” level=info msg=“[file datasource] opening file ‘/var/log/auth.log’”
time=“22-03-2021 16:55:33” level=info msg=“[file datasource] opening file ‘/var/log/syslog’”
time=“22-03-2021 16:55:33” level=info msg=“[file datasource] opening file ‘/var/log/kern.log’”
time=“22-03-2021 16:55:33” level=warning msg=“while configuring datasource : empty filename(s) and journalctl filter, malformed datasource”
time=“22-03-2021 16:55:33” level=info msg=“test done”
time=“22-03-2021 16:55:34” level=info msg=“Crowdsec v1.0.9-a8b16a66b110ebe03bb330cda2600226a3a862d7”
time=“22-03-2021 16:55:34” level=info msg=“Loading CAPI pusher”
time=“22-03-2021 16:55:34” level=info msg=“Loading prometheus collectors”
time=“22-03-2021 16:55:34” level=info msg=“start crowdsec api push (interval: 30s)”
time=“22-03-2021 16:55:34” level=info msg=“start crowdsec api pull (interval: 2h)”
time=“22-03-2021 16:55:34” level=info msg=“start crowdsec api send metrics (interval: 30m)”
time=“22-03-2021 16:55:34” level=info msg=“Loading grok library /etc/crowdsec//patterns/”
time=“22-03-2021 16:55:36” level=info msg=“pull top: added 70 entries”
time=“22-03-2021 16:55:36” level=info msg=“Loading enrich plugins”
time=“22-03-2021 16:55:36” level=info msg=“Loading parsers 7 stages”
time=“22-03-2021 16:55:36” level=info msg=“Node in /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml has no name,author or description. Skipping.”
time=“22-03-2021 16:55:36” level=info msg=“Loaded 2 parser nodes” file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
time=“22-03-2021 16:55:36” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s01-parse/nginx-logs.yaml
time=“22-03-2021 16:55:36” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
time=“22-03-2021 16:55:36” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
time=“22-03-2021 16:55:36” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
time=“22-03-2021 16:55:36” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s02-enrich/http-logs.yaml
time=“22-03-2021 16:55:36” level=info msg=“Loaded 1 parser nodes” file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml
time=“22-03-2021 16:55:36” level=info msg=“Loaded 8 nodes, 3 stages”
time=“22-03-2021 16:55:36” level=info msg=“Loading postoverflow Parsers”
time=“22-03-2021 16:55:36” level=info msg=“Loaded 0 nodes, 0 stages”
time=“22-03-2021 16:55:36” level=info msg=“Loading 11 scenario files”
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=young-pond file=/etc/crowdsec/scenarios/http-generic-bf.yaml name=crowdsecurity/http-generic-bf
time=“22-03-2021 16:55:36” level=info msg=“Adding trigger bucket” cfg=late-wind file=/etc/crowdsec/scenarios/http-w00tw00t.yaml name=ltsich/http-w00tw00t
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=billowing-pine file=/etc/crowdsec/scenarios/http-path-traversal-probing.yaml name=crowdsecurity/http-path-traversal-probing
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=bitter-snow file=/etc/crowdsec/scenarios/http-xss-probing.yaml name=crowdsecurity/http-xss-probbing
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=wild-fire file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=lingering-smoke file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=wispy-snowflake file=/etc/crowdsec/scenarios/http-sensitive-files.yaml name=crowdsecurity/http-sensitive-files
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=white-butterfly file=/etc/crowdsec/scenarios/http-sqli-probing.yaml name=crowdsecurity/http-sqli-probbing-detection
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=throbbing-bird file=/etc/crowdsec/scenarios/http-backdoors-attempts.yaml name=crowdsecurity/http-backdoors-attempts
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=green-wood file=/etc/crowdsec/scenarios/http-bad-user-agent.yaml name=crowdsecurity/http-bad-user-agent
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=fragrant-water file=/etc/crowdsec/scenarios/http-crawl-non_statics.yaml name=crowdsecurity/http-crawl-non_statics
time=“22-03-2021 16:55:36” level=info msg=“Adding leaky bucket” cfg=wild-sun file=/etc/crowdsec/scenarios/http-probing.yaml name=crowdsecurity/http-probing
time=“22-03-2021 16:55:36” level=warning msg=“Loaded 12 scenarios”
time=“22-03-2021 16:55:36” level=info msg=“loading acquisition file : /etc/crowdsec/acquis.yaml”
time=“22-03-2021 16:55:36” level=info msg=“[file datasource] opening file ‘/opt/myapp/logs/error.log’”
time=“22-03-2021 16:55:36” level=info msg=“[file datasource] opening file ‘/opt/myapp/logs/access.log’”
time=“22-03-2021 16:55:36” level=info msg=“[file datasource] opening file ‘/var/log/auth.log’”
time=“22-03-2021 16:55:36” level=info msg=“[file datasource] opening file ‘/var/log/syslog’”
time=“22-03-2021 16:55:36” level=info msg=“[file datasource] opening file ‘/var/log/kern.log’”
time=“22-03-2021 16:55:36” level=warning msg=“while configuring datasource : empty filename(s) and journalctl filter, malformed datasource”
time=“22-03-2021 16:55:36” level=warning msg=“Starting processing data”
Hello,
From these logs, it seems that those files are correctly opened. Can you show again what cscli metrics showed ? We should see some lines in acquisition about your “myapp” logs !
magic xD
:/etc/crowdsec# cscli metrics
INFO[22-03-2021 06:10:55 PM] Buckets Metrics:
±-------------------------------±--------------±----------±-------------±-------±--------+
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
±-------------------------------±--------------±----------±-------------±-------±--------+
| crowdsecurity/ssh-bf | 14 | - | 337 | 482 | 323 |
| crowdsecurity/ssh-bf_user-enum | 14 | - | 337 | 337 | 323 |
±-------------------------------±--------------±----------±-------------±-------±--------+
INFO[22-03-2021 06:10:55 PM] Acquisition Metrics:
±------------------±-----------±-------------±---------------±-----------------------+
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
±------------------±-----------±-------------±---------------±-----------------------+
| /var/log/auth.log | 1615 | 482 | 1133 | 819 |
| /var/log/syslog | 5 | - | 5 | - |
±------------------±-----------±-------------±---------------±-----------------------+
INFO[22-03-2021 06:10:55 PM] Parser Metrics:
±-------------------------------±-----±-------±---------+
| PARSERS | HITS | PARSED | UNPARSED |
±-------------------------------±-----±-------±---------+
| child-crowdsecurity/sshd-logs | 7250 | 482 | 6768 |
| crowdsecurity/dateparse-enrich | 482 | 482 | - |
| crowdsecurity/geoip-enrich | 482 | 482 | - |
| crowdsecurity/sshd-logs | 1606 | 482 | 1124 |
| crowdsecurity/syslog-logs | 1620 | 1620 | - |
| crowdsecurity/whitelists | 482 | 482 | - |
±-------------------------------±-----±-------±---------+
INFO[22-03-2021 06:10:55 PM] Local Api Metrics:
±-------------------±-------±-----+
| ROUTE | METHOD | HITS |
±-------------------±-------±-----+
| /v1/watchers/login | POST | 2 |
±-------------------±-------±-----+
From your paste, it seems that no logs were read from those files at all since crowdsec started, does it sounds plausible ? 
no, there are entries in access.log and error.log
Hi @Fleedstix, did you try to generate some new logs in access.log and error.log and check the metrics again ?