Help: having only unparsed logs

tcpip · March 5, 2021, 7:35am

Hey,

I have one system which seems not to parse any logs. It also does not show any decissions and on alterts only thing there is the update of the community rulesets. So looks like it is really not doing anything at the moment. I tried to find some clues but other similar threads mentioned in the Forum didn’t show any solution. Maybe just a minor Problem.

cscli metrics

INFO[0000] Acquisition Metrics:
+-----------------------------------+------------+--------------+----------------+------------------------+
|              SOURCE               | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log                 |        173 | -            |            173 | -                      |
| /var/log/caddy/dav-mgz.access.log |          1 | -            |              1 | -                      |
| /var/log/caddy/ip3q.access.log    |         16 | -            |             16 | -                      |
| /var/log/caddy/mgz.access.log     |        255 | -            |            255 | -                      |
| /var/log/syslog                   |       8484 | -            |           8484 | -                      |
+-----------------------------------+------------+--------------+----------------+------------------------+
INFO[0000] Parser Metrics:
+----------------------------------+------+--------+----------+
|             PARSERS              | HITS | PARSED | UNPARSED |
+----------------------------------+------+--------+----------+
| child-crowdsecurity/apache2-logs |  544 | -      |      544 |
| child-crowdsecurity/sshd-logs    |   65 | -      |       65 |
| crowdsecurity/apache2-logs       |  272 | -      |      272 |
| crowdsecurity/non-syslog         |  272 |    272 | -        |
| crowdsecurity/sshd-logs          |   13 | -      |       13 |
| crowdsecurity/syslog-logs        | 8657 |   8657 | -        |
+----------------------------------+------+--------+----------+
INFO[0000] Local Api Metrics:
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    |    1 |
| /v1/decisions/stream | GET    | 1062 |
| /v1/watchers/login   | POST   |    7 |
+----------------------+--------+------+
INFO[0000] Local Api Machines Metrics:
+----------------------------------+------------+--------+------+
|             MACHINE              |   ROUTE    | METHOD | HITS |
+----------------------------------+------------+--------+------+
| xxx| /v1/alerts | GET    |    1 |
+----------------------------------+------------+--------+------+
INFO[0000] Local Api Bouncers Metrics:
+------------------------------+----------------------+--------+------+
|           BOUNCER            |        ROUTE         | METHOD | HITS |
+------------------------------+----------------------+--------+------+
| cs-firewall-bouncer-SethwuNZ | /v1/decisions/stream | GET    | 1062 |
+------------------------------+----------------------+--------+------+

cscli parsers list

-------------------------------------------------------------------------------------------------------------
 NAME                            📦 STATUS   VERSION  LOCAL PATH
-------------------------------------------------------------------------------------------------------------
 crowdsecurity/iptables-logs     ✔️  enabled  0.2      /etc/crowdsec/parsers/s01-parse/iptables-logs.yaml
 crowdsecurity/apache2-logs      ✔️  enabled  0.4      /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml
 crowdsecurity/dateparse-enrich  ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
 crowdsecurity/http-logs         ✔️  enabled  0.5      /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
 crowdsecurity/whitelists        ✔️  enabled  0.1      /etc/crowdsec/parsers/s02-enrich/whitelists.yaml
 crowdsecurity/geoip-enrich      ✔️  enabled  0.2      /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
 crowdsecurity/mysql-logs        ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/mysql-logs.yaml
 crowdsecurity/sshd-logs         ✔️  enabled  0.1      /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
 crowdsecurity/syslog-logs       ✔️  enabled  0.1      /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
-------------------------------------------------------------------------------------------------------------

cscli scenarios list

--------------------------------------------------------------------------------------------------------------------------
 NAME                                       📦 STATUS   VERSION  LOCAL PATH
--------------------------------------------------------------------------------------------------------------------------
 crowdsecurity/http-path-traversal-probing  ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-path-traversal-probing.yaml
 crowdsecurity/http-probing                 ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-probing.yaml
 crowdsecurity/mysql-bf                     ✔️  enabled  0.1      /etc/crowdsec/scenarios/mysql-bf.yaml
 crowdsecurity/http-bad-user-agent          ✔️  enabled  0.3      /etc/crowdsec/scenarios/http-bad-user-agent.yaml
 crowdsecurity/http-crawl-non_statics       ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-crawl-non_statics.yaml
 crowdsecurity/http-sensitive-files         ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sensitive-files.yaml
 crowdsecurity/ssh-bf                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/ssh-bf.yaml
 crowdsecurity/http-backdoors-attempts      ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-backdoors-attempts.yaml
 crowdsecurity/http-sqli-probing            ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-sqli-probing.yaml
 ltsich/http-w00tw00t                       ✔️  enabled  0.1      /etc/crowdsec/scenarios/http-w00tw00t.yaml
 crowdsecurity/iptables-scan-multi_ports    ✔️  enabled  0.1      /etc/crowdsec/scenarios/iptables-scan-multi_ports.yaml
 crowdsecurity/http-xss-probing             ✔️  enabled  0.2      /etc/crowdsec/scenarios/http-xss-probing.yaml
--------------------------------------------------------------------------------------------------------------------------

cscli collections list

----------------------------------------------------------------------------------------------------------------
 NAME                                 📦 STATUS   VERSION  LOCAL PATH
----------------------------------------------------------------------------------------------------------------
 crowdsecurity/apache2                ✔️  enabled  0.1      /etc/crowdsec/collections/apache2.yaml
 crowdsecurity/base-http-scenarios    ✔️  enabled  0.3      /etc/crowdsec/collections/base-http-scenarios.yaml
 crowdsecurity/whitelist-good-actors  ✔️  enabled  0.1      /etc/crowdsec/collections/whitelist-good-actors.yaml
 crowdsecurity/sshd                   ✔️  enabled  0.1      /etc/crowdsec/collections/sshd.yaml
 crowdsecurity/linux                  ✔️  enabled  0.2      /etc/crowdsec/collections/linux.yaml
 crowdsecurity/mysql                  ✔️  enabled  0.1      /etc/crowdsec/collections/mysql.yaml
 crowdsecurity/iptables               ✔️  enabled  0.1      /etc/crowdsec/collections/iptables.yaml
----------------------------------------------------------------------------------------------------------------

In the logs there is no error.

Thanks for any Idea

thibault · March 5, 2021, 7:51am

Hello @tcpip !

Welcome and let’s look into this

From your metrics (the acquisition section) it seems that you are ingesting mostly caddy logs, and they aren’t parsed. I’m not too familiar with the log format, but it seems that apache2 tried and didn’t manage to parse them. You have as well some syslog and auth.log that aren’t parsed, but depending on your setup it might be normal.

Would you mind posting some (preferably anonymized) logs here so we can check if why it isn’t parsed by apache2 parsers ? (you can message me privately if you don’t feel comfortable posting it here).

Cheers,

tcpip · March 5, 2021, 10:32am

Yes its mostly Caddy and SSHD I want to parse.
I setup Caddy to use common apache2 log format here some examples:

116.203.xxx.xxx - - [05/Mar/2021:09:01:38 +0100] "GET /test/lists/sip_30d.txt HTTP/1.0" 200 1513
2a01:4f8:xxx:xxx:xxx::1 - - [05/Mar/2021:09:30:01 +0100] "GET /test/lists/sip_30d.txt.gz HTTP/1.0" 200 1523
116.203.xxx.xxx - - [05/Mar/2021:10:01:02 +0100] "GET /test/lists/sip_30d.txt.gz HTTP/2.0" 200 1498
116.203.xxx.xxx - - [05/Mar/2021:11:01:01 +0100] "GET /test/lists/sip_30d.txt.gz HTTP/2.0" 200 1498
83.151.xxx.xxx - - [05/Mar/2021:11:16:38 +0100] "POST /test/test.php HTTP/1.1" 200 0
63.143.xxx.xxx - - [05/Mar/2021:11:26:22 +0100] "HEAD / HTTP/1.1" 200 0
93.240.xxx.xxx - - [05/Mar/2021:11:27:29 +0100] "GET / HTTP/2.0" 200 1072
93.240.xxx.xxx - - [05/Mar/2021:11:27:29 +0100] "GET /favicon.ico HTTP/2.0" 404 0
93.240.xxx.xxx - - [05/Mar/2021:11:27:39 +0100] "GET / HTTP/2.0" 200 1072
93.240.xxx.xxx - - [05/Mar/2021:11:27:39 +0100] "GET /favicon.ico HTTP/2.0" 404 0
93.240.xxx.xxx - - [05/Mar/2021:11:30:18 +0100] "GET / HTTP/2.0" 401 0
93.240.xxx.xxx - - [05/Mar/2021:11:30:19 +0100] "GET / HTTP/2.0" 401 0
93.240.xxx.xxx - - [05/Mar/2021:11:30:43 +0100] "GET / HTTP/2.0" 401 0
93.240.xxx.xxx - - [05/Mar/2021:11:30:47 +0100] "GET / HTTP/2.0" 401 0
93.240.xxx.xxx - - [05/Mar/2021:11:30:52 +0100] "GET / HTTP/2.0" 401 0
93.240.xxx.xxx - - [05/Mar/2021:11:30:53 +0100] "GET / HTTP/2.0" 401 0
93.240.xxx.xxx - - [05/Mar/2021:11:30:55 +0100] "GET / HTTP/2.0" 401 0
93.240.xxx.xxx - - [05/Mar/2021:11:30:55 +0100] "GET / HTTP/2.0" 401 0

The strange thing is that from the metrics none of the log lines is parsed. Even mysql which is pretty standard everything not parsing.

+-----------------------------------+------------+--------------+----------------+------------------------+
|              SOURCE               | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log                 |        542 | -            |            542 | -                      |
| /var/log/caddy/dav-mgz.access.log |         34 | -            |             34 | -                      |
| /var/log/caddy/ip3q.access.log    |         19 | -            |             19 | -                      |
| /var/log/caddy/mgz.access.log     |        355 | -            |            355 | -                      |
| /var/log/mysql/error.log          |          2 | -            |              2 | -                      |
| /var/log/syslog                   |      11881 | -            |          11881 | -                      |
+-----------------------------------+------------+--------------+----------------+------------------------+

Some other posts suggest to check the dependencies but I went throu it and all are installed as far as I see.

If I try to process the file manually I only receive this:

crowdsec -file mgz.access.log -type apache2

INFO[05-03-2021 12:17:50] [file datasource] opening file 'mgz.access.log'
ERRO[05-03-2021 12:17:50] Failed to notify(sent: false): <nil>
WARN[05-03-2021 12:17:50] Starting processing data
INFO[05-03-2021 12:17:50] reading mgz.access.log at once
127.0.0.1 - [Fri, 05 Mar 2021 12:17:51 CET] "POST /v1/watchers/login HTTP/1.1 200 237.414315ms "crowdsec/v1.0.7-18ff3a3a306d1xxx" "
WARN[05-03-2021 12:17:51] Acquisition is finished, shutting down
INFO[05-03-2021 12:17:51] Killing parser routines
INFO[05-03-2021 12:17:51] Bucket routine exiting

thibault · March 5, 2021, 4:35pm

Hey !

I just checked, and I think our current apache2 log parser is a bit too restrictive and expects your logs to have the user-agent for example.

Would you mind trying the solution described in apache2 : parser is too restrictive · Issue #157 · crowdsecurity/hub · GitHub and see if it improves parsing capabilities ?

I will try to upstream this change soon

tcpip · March 5, 2021, 7:28pm

Thanks alot, for the Caddy Log this did the trick. It’s parsed now.

INFO[0000] Acquisition Metrics:
+-----------------------------------+------------+--------------+----------------+------------------------+
|              SOURCE               | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+-----------------------------------+------------+--------------+----------------+------------------------+
| /var/log/auth.log                 |         16 | -            |             16 | -                      |
| /var/log/caddy/dav-mgz.access.log |          8 |            8 | -              |                      3 |
| /var/log/caddy/mgz.access.log     |         11 |           11 | -              |                     10 |
| /var/log/syslog                   |         98 | -            |             98 | -                      |
+-----------------------------------+------------+--------------+----------------+------------------------+

Another Question is why the SSH logs are unparsed. Even if I manually cause an auth error it’s not hitting.

#/var/log/auth.log
Mar  5 20:24:14 testhost sshd[455340]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=178.200.xxx.xxx  user=root
Mar  5 20:24:17 testhost sshd[455336]: error: PAM: Authentication failure for root from 178.200.xxx.xxx

thibault · March 8, 2021, 8:01am

Hello,

I have opened the MR to make the [change of apache2 upstream](https://git hub.com/crowdsecurity/hub/pull/176).
I will try to look into the ssh stuff a bit later, would you mind sharing a few more log sample ?

Thanks,

thibault · March 9, 2021, 3:22pm

and hello again, I opened an issue to keep track of the sshd logs issue

thibault · April 8, 2021, 9:34am

Hello, this has been merged : add grok to catch auth fail log by he2ss · Pull Request #188 · crowdsecurity/hub · GitHub

Please let me know if it fixes your issue

Topic		Replies	Views
Help: caddy-logs parser failure crowdsec	5	1654	October 23, 2024
Question about how to read the metrics	11	3288	January 27, 2021
Cscli metrics unparsed logs crowdsec	4	1150	December 15, 2020
Nginx logs not being parsed	6	1351	September 15, 2021
Crowdsec Logs are not parsed	1	302	April 19, 2024

Help: having only unparsed logs

Related topics