Http requests like "\x16\x03\x01\x01\x22"

crowdsec
4

Hi @apass,

as the nginx-parser doesnt parse your log, you therefore need your own parser there. You need to correct two things in your custom parser,
first you should change the filter to “evt.Parsed.program startsWith ‘nginx’” and the apply_on of the grok node to message. After the non-syslog parser parsed your log-message, it is stored into the message field, wich you therefore need to apply the grok on.

All in all your custom Parser should look like this:

filter: "evt.Parsed.program startsWith 'nginx'"
onsuccess: next_stage
name: whoami/x00
description: "Parse http log for request like \x16\x03\x01\x01\x22"
#for clarity, we create our pattern syntax beforehand
pattern_syntax:
        x00_REQ: '%{IPORHOST:remote_addr} - %{NGUSER:remote_user} \[%{HTTPDATE:time_local}\] "\\%{DATA:raw_request}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"'

nodes:
#and we use them to parse our type of logs
  - grok:
      name: "x00_REQ"
      apply_on: message
      statics:
        - meta: log_type
          value: x00_bad_req
        - target: evt.StrTime
          expression: evt.Parsed.time_local

statics:
    - meta: service
      value: httpd
    - meta: source_ip
      expression: "evt.Parsed.remote_addr"
    - meta: http_status
      expression: "evt.Parsed.status"

But for inclusion in the hub we should correct the nginx-parser to act like the apache-parser. Then both parsers will parse the rawmessage, which we then can use inside a scenario to detect those attacks.

Modified Nginx-Parser, I added the rawrequest possibility

filter: "evt.Parsed.program startsWith 'nginx'"
onsuccess: next_stage
name: crowdsecurity/nginx-logs
description: "Parse nginx access and error logs"
nodes:
  - grok:
      pattern: '(%{IPORHOST:target_fqdn} )?%{IPORHOST:remote_addr} - (%{NGUSER:remote_user})? \[%{HTTPDATE:time_local}\] "((%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version})|%{DATA:rawrequest})" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"( %{NUMBER:request_length} %{NUMBER:request_time} \[%{DATA:proxy_upstream_name}\] \[%{DATA:proxy_alternative_upstream_name}\])?'
      apply_on: message
      statics:
        - meta: log_type
          value: http_access-log
        - target: evt.StrTime
          expression: evt.Parsed.time_local
  - grok:
        # and this one the error log
        pattern: '(%{IPORHOST:target_fqdn} )?%{NGINXERRTIME:time} \[%{LOGLEVEL:loglevel}\] %{NONNEGINT:pid}#%{NONNEGINT:tid}: (\*%{NONNEGINT:cid} )?%{GREEDYDATA:message}, client: %{IPORHOST:remote_addr}, server: %{DATA:target_fqdn}, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}", host: "%{IPORHOST}"'
        apply_on: message
        statics:
          - meta: log_type
            value: http_error-log
          - target: evt.StrTime
            expression: evt.Parsed.time
    pattern_syntax:
      NO_DOUBLE_QUOTE: '[^"]+'
    onsuccess: next_stage
    nodes:
      - filter: "evt.Parsed.message contains 'was not found in'"
        pattern_syntax:
          USER_NOT_FOUND: 'user "%{NO_DOUBLE_QUOTE:username}" was not found in "%{NO_DOUBLE_QUOTE}"'
        grok:
          pattern: '%{USER_NOT_FOUND}'
          apply_on: message
        statics:
          - meta: sub_type
            value: "auth_fail"
          - meta: username
            expression: evt.Parsed.username
      - filter: "evt.Parsed.message contains 'password mismatch'"
        pattern_syntax:
          PASSWORD_MISMATCH: 'user "%{NO_DOUBLE_QUOTE:username}": password mismatch'
        grok:
          pattern: '%{PASSWORD_MISMATCH}'
          apply_on: message
        statics:
          - meta: sub_type
            value: "auth_fail"
          - meta: username
            expression: evt.Parsed.username
    # these ones apply for both grok patterns
statics:
  - meta: service
    value: http
  - meta: source_ip
    expression: "evt.Parsed.remote_addr"
  - meta: http_status
    expression: "evt.Parsed.status"
  - meta: http_path
    expression: "evt.Parsed.request"
  - meta: http_verb
    expression: "evt.Parsed.verb"
  - meta: http_user_agent
    expression: "evt.Parsed.http_user_agent"
  - meta: target_fqdn
    expression: "evt.Parsed.target_fqdn"

New Scenario to detect your Events on both nginx and apache;

type: leaky
#debug: true
name: Dominic-Wagner/x00
description: "Detect bad requests"
filter: 'evt.Meta.log_type == "http_access-log" and evt.Parsed.rawrequest matches "\\\\x.*"'
capacity: 1
leakspeed: 1m
groupby: evt.Meta.source_ip
blackhole: 2m
reprocess: true
labels:
  service: http
  type: scan
  remediation: true

To make a PR on the hub and especially to use your logfile with scenarios you need to change your ip to something realistic (max 255.255.255.255, .444 is not a valid octet), else the scenario will not detect your events.

Best reguards
Dominic