Issue with own parser

crowdsec
1

When running sudo cscli explain -f /tmp/explain.txt --type syslog --verbose --debug -o raw > /tmp/explain-results.txt I can see that my own parser is failed (I am referring to g00g1/postfix-logs-milter):

cscli explain .... 
line: May 25 10:45:54 flopster postfix/cleanup[2346]: 8A96916E0059: milter-reject: END-OF-MESSAGE from out1.vger.email[2620:137:e000::1:20]: 5.7.1 mail 1GXasG from 2620:137:e000::1:20 rejected by DCC; from=<netfilter-owner@vger.kernel.org> to=<seentr@at.encryp.ch> proto=ESMTP helo=<out1.vger.email>
        ├ s00-raw
        |       └ 🟢 crowdsecurity/syslog-logs (+12 ~9)
        |               └ update evt.ExpectMode : %!s(int=0) -> 1
        |               └ update evt.Stage :  -> s01-parse
        |               └ update evt.Line.Raw :  -> May 25 10:45:54 flopster postfix/cleanup[2346]: 8A96916E0059: milter-reject: END-OF-MESSAGE from out1.vger.email[2620:137:e000::1:20]: 5.7.1 mail 1GXasG from 2620:137:e000::1:20 rejected by DCC; from=<netfilter-owner@vger.kernel.org> to=<seentr@at.encryp.ch> proto=ESMTP helo=<out1.vger.email>
        |               └ update evt.Line.Src :  -> /tmp/explain.txt
        |               └ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-05-26 09:57:46.110462548 +0000 UTC
        |               └ create evt.Line.Labels.type : syslog
        |               └ update evt.Line.Process : %!s(bool=false) -> true
        |               └ update evt.Line.Module :  -> file
        |               └ create evt.Parsed.facility :
        |               └ create evt.Parsed.logsource : syslog
        |               └ create evt.Parsed.message : 8A96916E0059: milter-reject: END-OF-MESSAGE from out1.vger.email[2620:137:e000::1:20]: 5.7.1 mail 1GXasG from 2620:137:e000::1:20 rejected by DCC; from=<netfilter-owner@vger.kernel.org> to=<seentr@at.encryp.ch> proto=ESMTP helo=<out1.vger.email>
        |               └ create evt.Parsed.pid : 2346
        |               └ create evt.Parsed.priority :
        |               └ create evt.Parsed.program : postfix/cleanup
        |               └ create evt.Parsed.timestamp : May 25 10:45:54
        |               └ create evt.Parsed.timestamp8601 :
        |               └ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-05-26 09:57:46.110673438 +0000 UTC
        |               └ update evt.StrTime :  -> May 25 10:45:54
        |               └ create evt.Meta.datasource_path : /tmp/explain.txt
        |               └ create evt.Meta.datasource_type : file
        |               └ create evt.Meta.machine : flopster
        ├ s01-parse
        |       ├ 🔴 crowdsecurity/dovecot-logs
        |       ├ 🔴 crowdsecurity/endlessh-logs
        |       ├ 🔴 crowdsecurity/nginx-logs
        |       ├ 🔴 crowdsecurity/postfix-logs
        |       ├ 🔴 crowdsecurity/postscreen-logs
        |       ├ 🔴 crowdsecurity/sshd-logs
        |       ├ 🔴 g00g1/nginx-custom-logs
        |       ├ 🔴 g00g1/postfix-logs-generic
        |       ├ 🔴 g00g1/postfix-logs-milter
        |       └ 🔴 g00g1/postfix-logs-smtpd
        └-------- parser failure 🔴
g00g1/postfix-logs-milter 
onsuccess: next_stage
debug: true
filter: "evt.Parsed.program == 'postfix/cleanup'"
name: g00g1/postfix-logs-milter
pattern_syntax:
  POSTFIX_QUEUEID: '([0-9A-F]{6,}|[0-9a-zA-Z]{12,}|NOQUEUE)' # https://github.com/cjslack/grok-debugger/blob/master/public/patterns/postfix
description: "Parse postfix logs (milter verdict only)"
nodes:
  - grok:
      apply_on: evt.Parsed.message
      pattern: '%{POSTFIX_QUEUEID:queue_id}: milter-reject: END-OF-MESSAGE from %{POSTFIX_HOSTNAME:remote_host}\[%{IP:remote_addr}\]: 5.\d.\d mail [0-9a-zA-Z]+ from %{IP} rejected by DCC; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=ESMTP helo=<%{HOSTNAME:helo}>'
      statics:
        - meta: log_type_enh
          value: g00g1-dcc-attempt
statics:
    - meta: service
      value: postfix
    - meta: queue_id
      expression: "evt.Parsed.queue_id"
    - meta: source_ip
      expression: "evt.Parsed.remote_addr"
    - meta: source_hostname
      expression: "evt.Parsed.remote_host"
    - meta: smtp_helo
      expression: "evt.Parsed.helo"
    - meta: smtp_from
      expression: "evt.Parsed.from"
    - meta: smtp_to
      expression: "evt.Parsed.to"
    - meta: log_type
      value: postfix

I have tested pattern %{POSTFIX_QUEUEID:queue_id}: milter-reject: END-OF-MESSAGE from %{POSTFIX_HOSTNAME:remote_host}\[%{IP:remote_addr}\]: 5.\d.\d mail [0-9a-zA-Z]+ from %{IP} rejected by DCC; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=ESMTP helo=<%{HOSTNAME:helo}> and payload 8A96916E0059: milter-reject: END-OF-MESSAGE from out1.vger.email[2620:137:e000::1:20]: 5.7.1 mail 1GXasG from 2620:137:e000::1:20 rejected by DCC; from=<netfilter-owner@vger.kernel.org> to=<seentr@at.encryp.ch> proto=ESMTP helo=<out1.vger.email> using grokdebugger.com and everything is parsed as I thought it should be. But crowdsec does not for some reason, and at this point I am unable to do further troubleshooting due to absence of more information in the documentation.

2

The apply_on just needs to be the key so it should just need to be

nodes:
  - grok:
      apply_on: message
      pattern: '%{POSTFIX_QUEUEID:queue_id}: milter-reject: END-OF-MESSAGE from %{POSTFIX_HOSTNAME:remote_host}\[%{IP:remote_addr}\]: 5.\d.\d mail [0-9a-zA-Z]+ from %{IP} rejected by DCC; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=ESMTP helo=<%{HOSTNAME:helo}>'
      statics:
        - meta: log_type_enh
          value: g00g1-dcc-attempt
1 Like
3

Just note this is not very clear in the docs that apply_on takes a key that is from evt.Parsed object. Will make it clearer