Hi,
sorry for the maybe easy question but I have a lot of request to apache/nginx in this form:
111.222.333.444 - - [11/Mar/2022:07:41:47 +0100] "\x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-"
How to write a grok to catch them?
Thanks in advance.
Hi,
so for writing a grok pattern to extract certain information, you should first look for the fields you want to extract.
So for your log-line i assume you want the ip, timestamp and raw_data and maybe the response code, and you only want to parse these complex requests that start with .
So your log looks like an nginx accesslog, but the attacker doesn’t use standard request methods like GET, HEAD or POST.
For that you need to correct the grok line to parse these complex requests. I therefore removed the method, request and http_version fields and replaced them with a single %{DATA:raw_request}.
The "\ before that is necessary to only parse logs starting with . As \ is a special character, we need to escape it with a \ before it.
%{IPORHOST:remote_addr} - %{NGUSER:remote_user} \[%{HTTPDATE:time_local}\] "\\%{DATA:raw_request}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"
Best reguards
Dominic
Hi @Dominic-Wagner, thanks for your reply.
This is my custom parser:
onsuccess: next_stage
filter: "evt.Meta.log_type == 'http_access-log'"
name: whoami/x00
description: "Parse http log for request like \x16\x03\x01\x01\x22"
#for clarity, we create our pattern syntax beforehand
pattern_syntax:
#x00_REQ: '.*\\x.*'
x00_REQ: '%{IPORHOST:remote_addr} - %{NGUSER:remote_user} \[%{HTTPDATE:time_local}\] "\\%{DATA:raw_request}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"'
nodes:
#and we use them to parse our type of logs
- grok:
name: "x00_REQ"
apply_on: raw_request
statics:
- meta: log_type
value: x00_bad_req
statics:
- meta: service
value: httpd
- meta: source_ip
expression: "evt.Parsed.source_ip"
but when I try to debug this kind of logs with type nginx I have a parser failure
sudo cscli explain --file ~/log_x00.log --type nginx -v
line: 111.222.333.444 - - [11/Mar/2022:07:41:47 +0100] "\x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| ├ 🔴 crowdsecurity/apache2-logs
| ├ 🔴 crowdsecurity/iptables-logs
| ├ 🔴 crowdsecurity/modsecurity
| ├ 🔴 crowdsecurity/pam-logs
| ├ 🔴 crowdsecurity/pkexec-logs
| ├ 🔴 crowdsecurity/postfix-logs
| ├ 🔴 crowdsecurity/postscreen-logs
| ├ 🔴 crowdsecurity/sshd-logs
| ├ 🔴 whoami/fd
| ├ 🔴 whoami/myssh
| └ 🔴 whoami/x00
└-------- parser failure 🔴
instead if I try to debug with a type apache2 it works but with another (default and not custom) parser:
sudo cscli explain --file ~/log_x00.log --type apache2 -v
line: 111.222.333.444 - - [11/Mar/2022:07:41:47 +0100] "\x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| └ 🟢 crowdsecurity/apache2-logs (+17 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.clientip : 111.222.333.444
| └ create evt.Parsed.httpversion :
| └ create evt.Parsed.referrer : -
| └ create evt.Parsed.response : 400
| └ create evt.Parsed.verb :
| └ create evt.Parsed.http_user_agent : -
| └ create evt.Parsed.request :
| └ create evt.Parsed.auth : -
| └ create evt.Parsed.ident : -
| └ create evt.Parsed.rawrequest : \x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0
| └ create evt.Parsed.target_fqdn :
| └ create evt.Parsed.timestamp : 11/Mar/2022:07:41:47 +0100
| └ create evt.Parsed.bytes : 157
| └ update evt.StrTime : -> 11/Mar/2022:07:41:47 +0100
| └ create evt.Meta.log_type : http_access-log
| └ create evt.Meta.service : http
| └ create evt.Meta.source_ip : 111.222.333.444
| └ create evt.Meta.http_status : 400
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2022-03-11T07:41:47+01:00
| ├ update evt.MarshaledTime : -> 2022-03-11T07:41:47+01:00
| ├ create evt.Meta.timestamp : 2022-03-11T07:41:47+01:00
| ├ 🟢 crowdsecurity/geoip-enrich (+10)
| ├ create evt.Enriched.ASNNumber : 0
| ├ create evt.Enriched.ASNOrg :
| ├ create evt.Enriched.ASNumber : 0
| ├ create evt.Enriched.IsInEU : false
| ├ create evt.Enriched.IsoCode : US
| ├ create evt.Enriched.Latitude : 29.966800
| ├ create evt.Enriched.Longitude : -95.345400
| ├ create evt.Meta.IsoCode : US
| ├ create evt.Meta.ASNNumber : 0
| ├ create evt.Meta.IsInEU : false
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/whitelists (+2)
| ├ create evt.Parsed.impact_completion : true
| ├ create evt.Parsed.static_ressource : false
| └ 🟢 whoami/wg-whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
└ 🟢 crowdsecurity/http-probing
What’s wrong with my custom parser? Or should I use the default (apache) one?
I’d like to catch this type of request in all machines where there is a web server (apache or nginx) because this kind of requests are attacks…
Hi @apass,
as the nginx-parser doesnt parse your log, you therefore need your own parser there. You need to correct two things in your custom parser,
first you should change the filter to “evt.Parsed.program startsWith ‘nginx’” and the apply_on of the grok node to message. After the non-syslog parser parsed your log-message, it is stored into the message field, wich you therefore need to apply the grok on.
All in all your custom Parser should look like this:
filter: "evt.Parsed.program startsWith 'nginx'"
onsuccess: next_stage
name: whoami/x00
description: "Parse http log for request like \x16\x03\x01\x01\x22"
#for clarity, we create our pattern syntax beforehand
pattern_syntax:
x00_REQ: '%{IPORHOST:remote_addr} - %{NGUSER:remote_user} \[%{HTTPDATE:time_local}\] "\\%{DATA:raw_request}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"'
nodes:
#and we use them to parse our type of logs
- grok:
name: "x00_REQ"
apply_on: message
statics:
- meta: log_type
value: x00_bad_req
- target: evt.StrTime
expression: evt.Parsed.time_local
statics:
- meta: service
value: httpd
- meta: source_ip
expression: "evt.Parsed.remote_addr"
- meta: http_status
expression: "evt.Parsed.status"
But for inclusion in the hub we should correct the nginx-parser to act like the apache-parser. Then both parsers will parse the rawmessage, which we then can use inside a scenario to detect those attacks.
Modified Nginx-Parser, I added the rawrequest possibility
filter: "evt.Parsed.program startsWith 'nginx'"
onsuccess: next_stage
name: crowdsecurity/nginx-logs
description: "Parse nginx access and error logs"
nodes:
- grok:
pattern: '(%{IPORHOST:target_fqdn} )?%{IPORHOST:remote_addr} - (%{NGUSER:remote_user})? \[%{HTTPDATE:time_local}\] "((%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version})|%{DATA:rawrequest})" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"( %{NUMBER:request_length} %{NUMBER:request_time} \[%{DATA:proxy_upstream_name}\] \[%{DATA:proxy_alternative_upstream_name}\])?'
apply_on: message
statics:
- meta: log_type
value: http_access-log
- target: evt.StrTime
expression: evt.Parsed.time_local
- grok:
# and this one the error log
pattern: '(%{IPORHOST:target_fqdn} )?%{NGINXERRTIME:time} \[%{LOGLEVEL:loglevel}\] %{NONNEGINT:pid}#%{NONNEGINT:tid}: (\*%{NONNEGINT:cid} )?%{GREEDYDATA:message}, client: %{IPORHOST:remote_addr}, server: %{DATA:target_fqdn}, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}", host: "%{IPORHOST}"'
apply_on: message
statics:
- meta: log_type
value: http_error-log
- target: evt.StrTime
expression: evt.Parsed.time
pattern_syntax:
NO_DOUBLE_QUOTE: '[^"]+'
onsuccess: next_stage
nodes:
- filter: "evt.Parsed.message contains 'was not found in'"
pattern_syntax:
USER_NOT_FOUND: 'user "%{NO_DOUBLE_QUOTE:username}" was not found in "%{NO_DOUBLE_QUOTE}"'
grok:
pattern: '%{USER_NOT_FOUND}'
apply_on: message
statics:
- meta: sub_type
value: "auth_fail"
- meta: username
expression: evt.Parsed.username
- filter: "evt.Parsed.message contains 'password mismatch'"
pattern_syntax:
PASSWORD_MISMATCH: 'user "%{NO_DOUBLE_QUOTE:username}": password mismatch'
grok:
pattern: '%{PASSWORD_MISMATCH}'
apply_on: message
statics:
- meta: sub_type
value: "auth_fail"
- meta: username
expression: evt.Parsed.username
# these ones apply for both grok patterns
statics:
- meta: service
value: http
- meta: source_ip
expression: "evt.Parsed.remote_addr"
- meta: http_status
expression: "evt.Parsed.status"
- meta: http_path
expression: "evt.Parsed.request"
- meta: http_verb
expression: "evt.Parsed.verb"
- meta: http_user_agent
expression: "evt.Parsed.http_user_agent"
- meta: target_fqdn
expression: "evt.Parsed.target_fqdn"
New Scenario to detect your Events on both nginx and apache;
type: leaky
#debug: true
name: Dominic-Wagner/x00
description: "Detect bad requests"
filter: 'evt.Meta.log_type == "http_access-log" and evt.Parsed.rawrequest matches "\\\\x.*"'
capacity: 1
leakspeed: 1m
groupby: evt.Meta.source_ip
blackhole: 2m
reprocess: true
labels:
service: http
type: scan
remediation: true
To make a PR on the hub and especially to use your logfile with scenarios you need to change your ip to something realistic (max 255.255.255.255, .444 is not a valid octet), else the scenario will not detect your events.
Best reguards
Dominic
Hi @Dominic-Wagner,
many thanks again for your help.
Everything is quite clear in you reply but non completely…
First: my custom whoami/x00 parser now works. Good.
Second: you wrote about a “modified nginx-parser” (crowdsecurity/nginx-logs) but:
Should I replace my crowdsecurity/nginx-logs with your modified or should I download and update it from the hub? Or
Should I use my custom whoami/x00 parser?
Third, I modified your scenario in this way but it does not work:
type: trigger
#debug: true
name: whoami/x00
description: "Detect bad requests like \x16\x03\x01\x01\x22"
filter: 'evt.Meta.log_type == "http_access-log" and evt.Parsed.rawrequest matches "\\\\x.*"'
#capacity: 1
#leakspeed: 1m
groupby: evt.Meta.source_ip
blackhole: 1m
reprocess: true
labels:
service: http
type: scan
remediation: true
To make a PR on the hub and especially to use your logfile with scenarios you need to change your ip to something realistic (max 255.255.255.255, .444 is not a valid octet), else the scenario will not detect your events.
What’s a “PR”?
The ip I posted in this topic is not real and the log contains the real IP but I replaced it before posting the log…
Thanks.
Hi @apass,
no problem.
Currently downloading an updating the ( crowdsecurity/nginx-logs ) parser wouldn’t help you with your Problem. As it yet doesn’t include the rawrequest field. I will open an Pull-Request on Github there tomorrow, so that it will be included in the hub in the future.
For now you can use your own parser. When that nginx-logs parser received those changes, you can just update and delete your own parser, but not for now.
To make your own parser work with that scenario, you will need to change the meta-information logtype on the grok node inside your parser.
Your parsers grok node should then look like the following:
- grok:
name: "x00_REQ"
apply_on: message
statics:
- meta: log_type
value: http_access-log
- target: evt.StrTime
expression: evt.Parsed.time_local
After that it also will be detected by that scenario.
PR stands for Pull-Request on GitHub.
The intention behind the crowdsec hub is, that everybody can submit parsers, scenarios and collection there for others to use. With a Pull-Request you can submit your scenario for other people to use it afterwards. To check that scenarios and parsers in the hub work, there need to be tests which will not work with invalid ips, like the .444.
Kind Regards
Hi @Dominic-Wagner ,
the scenario works in a machine with apache but it doen’t in a machine with nginx.
This is the log that I’m trying with (I modified all the IP with fake addresses):
111.111.111.111 - - [11/Mar/2022:07:41:47 +0100] "\x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-"
222.222.222.222 - - [20/Mar/2022:21:16:56 +0100] "\x16\x03\x01\x01\xfa\x01" 400 0 "-" "-"
333.333.333.333 - - [20/Mar/2022:23:28:38 +0100] "\x16\x03\x01" 400 0 "-" "-"
This is the test with apache logtype:
sudo cscli explain --file ~/log_x00.log --type apache2 -v
line: 333.333.333.333 - - [20/Mar/2022:23:28:38 +0100] "\x16\x03\x01" 400 0 "-" "-"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| └ 🟢 crowdsecurity/apache2-logs (+17 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.auth : -
| └ create evt.Parsed.target_fqdn :
| └ create evt.Parsed.response : 400
| └ create evt.Parsed.clientip : 333.333.333.333
| └ create evt.Parsed.httpversion :
| └ create evt.Parsed.request :
| └ create evt.Parsed.timestamp : 20/Mar/2022:23:28:38 +0100
| └ create evt.Parsed.verb :
| └ create evt.Parsed.referrer : -
| └ create evt.Parsed.bytes : 0
| └ create evt.Parsed.http_user_agent : -
| └ create evt.Parsed.ident : -
| └ create evt.Parsed.rawrequest : \x16\x03\x01
| └ update evt.StrTime : -> 20/Mar/2022:23:28:38 +0100
| └ create evt.Meta.http_status : 400
| └ create evt.Meta.log_type : http_access-log
| └ create evt.Meta.service : http
| └ create evt.Meta.source_ip : 333.333.333.333
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2022-03-20T23:28:38+01:00
| ├ update evt.MarshaledTime : -> 2022-03-20T23:28:38+01:00
| ├ create evt.Meta.timestamp : 2022-03-20T23:28:38+01:00
| ├ 🟢 crowdsecurity/geoip-enrich (+13)
| ├ create evt.Enriched.ASNNumber : 6428
| ├ create evt.Enriched.ASNOrg : CDM
| ├ create evt.Enriched.ASNumber : 6428
| ├ create evt.Enriched.IsInEU : false
| ├ create evt.Enriched.IsoCode : US
| ├ create evt.Enriched.Longitude : -90.501600
| ├ create evt.Enriched.Latitude : 38.710000
| ├ create evt.Enriched.SourceRange : 333.333.333.0/18
| ├ create evt.Meta.ASNNumber : 6428
| ├ create evt.Meta.ASNOrg : CDM
| ├ create evt.Meta.IsInEU : false
| ├ create evt.Meta.IsoCode : US
| ├ create evt.Meta.SourceRange : 333.333.333.0/18
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/whitelists (+2)
| ├ create evt.Parsed.impact_completion : true
| ├ create evt.Parsed.static_ressource : false
| └ 🟢 whoami/wg-whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
├ 🟢 crowdsecurity/http-probing
└ 🟢 whoami/x00
line:
├ s00-enrich
| └ 🟢 crowdsecurity/rdns (first_parser)
├ s01-whitelist
| ├ 🟢 crowdsecurity/cdn-whitelist (unchanged)
| └ 🟢 crowdsecurity/seo-bots-whitelist (unchanged)
└-------- parser failure 🔴
line: 111.111.111.111 - - [11/Mar/2022:07:41:47 +0100] "\x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| └ 🟢 crowdsecurity/apache2-logs (+17 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.bytes : 157
| └ create evt.Parsed.clientip : 111.111.111.111
| └ create evt.Parsed.ident : -
| └ create evt.Parsed.request :
| └ create evt.Parsed.timestamp : 11/Mar/2022:07:41:47 +0100
| └ create evt.Parsed.auth : -
| └ create evt.Parsed.referrer : -
| └ create evt.Parsed.http_user_agent : -
| └ create evt.Parsed.httpversion :
| └ create evt.Parsed.response : 400
| └ create evt.Parsed.rawrequest : \x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0
| └ create evt.Parsed.target_fqdn :
| └ create evt.Parsed.verb :
| └ update evt.StrTime : -> 11/Mar/2022:07:41:47 +0100
| └ create evt.Meta.http_status : 400
| └ create evt.Meta.log_type : http_access-log
| └ create evt.Meta.service : http
| └ create evt.Meta.source_ip : 111.111.111.111
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2022-03-11T07:41:47+01:00
| ├ update evt.MarshaledTime : -> 2022-03-11T07:41:47+01:00
| ├ create evt.Meta.timestamp : 2022-03-11T07:41:47+01:00
| ├ 🟢 crowdsecurity/geoip-enrich (+10)
| ├ create evt.Enriched.IsoCode : US
| ├ create evt.Enriched.Latitude : 29.966800
| ├ create evt.Enriched.Longitude : -95.345400
| ├ create evt.Enriched.ASNNumber : 0
| ├ create evt.Enriched.ASNOrg :
| ├ create evt.Enriched.ASNumber : 0
| ├ create evt.Enriched.IsInEU : false
| ├ create evt.Meta.IsoCode : US
| ├ create evt.Meta.ASNNumber : 0
| ├ create evt.Meta.IsInEU : false
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/whitelists (+2)
| ├ create evt.Parsed.impact_completion : true
| ├ create evt.Parsed.static_ressource : false
| └ 🟢 whoami/wg-whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
├ 🟢 crowdsecurity/http-probing
└ 🟢 whoami/x00
line: 222.222.222.222 - - [20/Mar/2022:21:16:56 +0100] "\x16\x03\x01\x01\xfa\x01" 400 0 "-" "-"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| └ 🟢 crowdsecurity/apache2-logs (+17 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.referrer : -
| └ create evt.Parsed.target_fqdn :
| └ create evt.Parsed.timestamp : 20/Mar/2022:21:16:56 +0100
| └ create evt.Parsed.bytes : 0
| └ create evt.Parsed.ident : -
| └ create evt.Parsed.auth : -
| └ create evt.Parsed.httpversion :
| └ create evt.Parsed.verb :
| └ create evt.Parsed.clientip : 222.222.222.222
| └ create evt.Parsed.http_user_agent : -
| └ create evt.Parsed.rawrequest : \x16\x03\x01\x01\xfa\x01
| └ create evt.Parsed.request :
| └ create evt.Parsed.response : 400
| └ update evt.StrTime : -> 20/Mar/2022:21:16:56 +0100
| └ create evt.Meta.http_status : 400
| └ create evt.Meta.log_type : http_access-log
| └ create evt.Meta.service : http
| └ create evt.Meta.source_ip : 222.222.222.222
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2022-03-20T21:16:56+01:00
| ├ update evt.MarshaledTime : -> 2022-03-20T21:16:56+01:00
| ├ create evt.Meta.timestamp : 2022-03-20T21:16:56+01:00
| ├ 🟢 crowdsecurity/geoip-enrich (+10)
| ├ create evt.Enriched.ASNNumber : 0
| ├ create evt.Enriched.ASNOrg :
| ├ create evt.Enriched.ASNumber : 0
| ├ create evt.Enriched.IsInEU : false
| ├ create evt.Enriched.IsoCode : US
| ├ create evt.Enriched.Latitude : 32.879500
| ├ create evt.Enriched.Longitude : -96.939800
| ├ create evt.Meta.ASNNumber : 0
| ├ create evt.Meta.IsInEU : false
| ├ create evt.Meta.IsoCode : US
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/whitelists (+2)
| ├ create evt.Parsed.static_ressource : false
| ├ create evt.Parsed.impact_completion : true
| └ 🟢 whoami/wg-whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
├ 🟢 crowdsecurity/http-probing
└ 🟢 whoami/x00
And this is the test with nginx logtype:
sudo cscli explain --file ~/log_x00.log --type nginx -v
line: 111.111.111.111 - - [11/Mar/2022:07:41:47 +0100] "\x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0" 400 157 "-" "-"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| ├ 🔴 crowdsecurity/apache2-logs
| ├ 🔴 crowdsecurity/iptables-logs
| ├ 🔴 crowdsecurity/modsecurity
| ├ 🔴 crowdsecurity/nginx-logs
| ├ 🔴 crowdsecurity/nginx-proxy-manager-logs
| ├ 🔴 crowdsecurity/pam-logs
| ├ 🔴 crowdsecurity/pkexec-logs
| ├ 🔴 crowdsecurity/postfix-logs
| ├ 🔴 crowdsecurity/postscreen-logs
| ├ 🔴 crowdsecurity/sshd-logs
| ├ 🔴 whoami/sshd-more-logs
| └ 🟢 whoami/x00 (+11 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.remote_user : -
| └ create evt.Parsed.body_bytes_sent : 157
| └ create evt.Parsed.remote_addr : 111.111.111.111
| └ create evt.Parsed.raw_request : "\x16\x03\x01\x00\xCA\x01\x00\x00\xC6\x03\x03\xC3\xA3\xF6MU\xBAZJ2\xBA\xD3\xCB\xAD\xA9\x92~j\x0E<\x8Cf,\xBB\x9A)\xD4\xAD53\xF3\x04\x0E\x00\x00h\xCC\x14\xCC\x13\xC0/\xC0+\xC00\xC0,\xC0\x11\xC0\x07\xC0'\xC0#\xC0\x13\xC0\x09\xC0(\xC0$\xC0\x14\xC0"
| └ create evt.Parsed.status : 400
| └ create evt.Parsed.time_local : 11/Mar/2022:07:41:47 +0100
| └ create evt.Parsed.http_referer : -
| └ create evt.Parsed.http_user_agent : -
| └ update evt.StrTime : -> 11/Mar/2022:07:41:47 +0100
| └ create evt.Meta.http_status : 400
| └ create evt.Meta.log_type : http_access-log
| └ create evt.Meta.service : httpd
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2022-03-11T07:41:47+01:00
| ├ update evt.MarshaledTime : -> 2022-03-11T07:41:47+01:00
| ├ create evt.Meta.timestamp : 2022-03-11T07:41:47+01:00
| ├ 🔴 crowdsecurity/geoip-enrich
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/whitelists (unchanged)
| └ 🟢 whoami/wg-whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
line: 222.222.222.222 - - [20/Mar/2022:21:16:56 +0100] "\x16\x03\x01\x01\xfa\x01" 400 0 "-" "-"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| ├ 🔴 crowdsecurity/apache2-logs
| ├ 🔴 crowdsecurity/iptables-logs
| ├ 🔴 crowdsecurity/modsecurity
| ├ 🔴 crowdsecurity/nginx-logs
| ├ 🔴 crowdsecurity/nginx-proxy-manager-logs
| ├ 🔴 crowdsecurity/pam-logs
| ├ 🔴 crowdsecurity/pkexec-logs
| ├ 🔴 crowdsecurity/postfix-logs
| ├ 🔴 crowdsecurity/postscreen-logs
| ├ 🔴 crowdsecurity/sshd-logs
| ├ 🔴 whoami/sshd-more-logs
| └ 🟢 whoami/x00 (+11 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.body_bytes_sent : 0
| └ create evt.Parsed.remote_user : -
| └ create evt.Parsed.status : 400
| └ create evt.Parsed.http_referer : -
| └ create evt.Parsed.http_user_agent : -
| └ create evt.Parsed.raw_request : "\x16\x03\x01\x01\xfa\x01"
| └ create evt.Parsed.remote_addr : 222.222.222.222
| └ create evt.Parsed.time_local : 20/Mar/2022:21:16:56 +0100
| └ update evt.StrTime : -> 20/Mar/2022:21:16:56 +0100
| └ create evt.Meta.service : httpd
| └ create evt.Meta.http_status : 400
| └ create evt.Meta.log_type : http_access-log
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2022-03-20T21:16:56+01:00
| ├ update evt.MarshaledTime : -> 2022-03-20T21:16:56+01:00
| ├ create evt.Meta.timestamp : 2022-03-20T21:16:56+01:00
| ├ 🔴 crowdsecurity/geoip-enrich
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/whitelists (unchanged)
| └ 🟢 whoami/wg-whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
line: 333.333.333.333 - - [20/Mar/2022:23:28:38 +0100] "\x16\x03\x01" 400 0 "-" "-"
├ s00-raw
| ├ 🟢 crowdsecurity/non-syslog (first_parser)
| └ 🔴 crowdsecurity/syslog-logs
├ s01-parse
| ├ 🔴 crowdsecurity/apache2-logs
| ├ 🔴 crowdsecurity/iptables-logs
| ├ 🔴 crowdsecurity/modsecurity
| ├ 🔴 crowdsecurity/nginx-logs
| ├ 🔴 crowdsecurity/nginx-proxy-manager-logs
| ├ 🔴 crowdsecurity/pam-logs
| ├ 🔴 crowdsecurity/pkexec-logs
| ├ 🔴 crowdsecurity/postfix-logs
| ├ 🔴 crowdsecurity/postscreen-logs
| ├ 🔴 crowdsecurity/sshd-logs
| ├ 🔴 whoami/sshd-more-logs
| └ 🟢 whoami/x00 (+11 ~2)
| └ update evt.Stage : s01-parse -> s02-enrich
| └ create evt.Parsed.remote_user : -
| └ create evt.Parsed.status : 400
| └ create evt.Parsed.body_bytes_sent : 0
| └ create evt.Parsed.http_referer : -
| └ create evt.Parsed.remote_addr : 333.333.333.333
| └ create evt.Parsed.http_user_agent : -
| └ create evt.Parsed.raw_request : "\x16\x03\x01"
| └ create evt.Parsed.time_local : 20/Mar/2022:23:28:38 +0100
| └ update evt.StrTime : -> 20/Mar/2022:23:28:38 +0100
| └ create evt.Meta.http_status : 400
| └ create evt.Meta.log_type : http_access-log
| └ create evt.Meta.service : httpd
├ s02-enrich
| ├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
| ├ create evt.Enriched.MarshaledTime : 2022-03-20T23:28:38+01:00
| ├ update evt.MarshaledTime : -> 2022-03-20T23:28:38+01:00
| ├ create evt.Meta.timestamp : 2022-03-20T23:28:38+01:00
| ├ 🔴 crowdsecurity/geoip-enrich
| ├ 🔴 crowdsecurity/http-logs
| ├ 🟢 crowdsecurity/whitelists (unchanged)
| └ 🟢 whoami/wg-whitelists (unchanged)
├-------- parser success 🟢
├ Scenarios
Why does it work with apache but not with nginx?
The scenario is the same:
type: trigger
#debug: true
name: whoami/x00
description: "Detect bad requests like \x16\x03\x01\x01\x22"
filter: 'evt.Meta.log_type == "http_access-log" and evt.Parsed.rawrequest matches "\\\\x.*"'
#capacity: 1
#leakspeed: 1m
groupby: evt.Meta.source_ip
blackhole: 1m
reprocess: true
labels:
service: http
type: scan
remediation: true
Many thanks again.
Hi @apass,
if you look closely at the cscli explain output, your own parser has the double quotes included, the apache parser not. Try adding them before and after the raw_request and remove the underscore from raw_request in your parser, as thats the field the scenario is looking for.
"%{DATA:rawrequest}"
Best Regards
Just an update this has now been integrated into the official NGINX parser so all requests will be parsed even if they are not an HTTP formatted request