# Issue with own parser

**URL:** <https://discourse.crowdsec.net/t/issue-with-own-parser/1306>\
**Category:** crowdsec\
**Created:** [May 26, 2023, 10:14am UTC](https://discourse.crowdsec.net/t/issue-with-own-parser/1306 "2023-05-26T10:14:25Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![g00g1](https://avatars.discourse-cdn.com/v4/letter/g/6a8cbe/32.png) [@g00g1](https://discourse.crowdsec.net/u/g00g1)\
**Post date:** [May 26, 2023, 10:14am UTC](https://discourse.crowdsec.net/t/issue-with-own-parser/1306/1 "2023-05-26T10:14:25Z")

</div>

When running `sudo cscli explain -f /tmp/explain.txt --type syslog --verbose --debug -o raw > /tmp/explain-results.txt` I can see that my own parser is failed (I am referring to `g00g1/postfix-logs-milter`):

> **cscli explain ....**
>
> ```auto
> line: May 25 10:45:54 flopster postfix/cleanup[2346]: 8A96916E0059: milter-reject: END-OF-MESSAGE from out1.vger.email[2620:137:e000::1:20]: 5.7.1 mail 1GXasG from 2620:137:e000::1:20 rejected by DCC; from=<netfilter-owner@vger.kernel.org> to=<seentr@at.encryp.ch> proto=ESMTP helo=<out1.vger.email>
> ├ s00-raw
> | └ 🟢 crowdsecurity/syslog-logs (+12 ~9)
> | └ update evt.ExpectMode : %!s(int=0) -> 1
> | └ update evt.Stage : -> s01-parse
> | └ update evt.Line.Raw : -> May 25 10:45:54 flopster postfix/cleanup[2346]: 8A96916E0059: milter-reject: END-OF-MESSAGE from out1.vger.email[2620:137:e000::1:20]: 5.7.1 mail 1GXasG from 2620:137:e000::1:20 rejected by DCC; from=<netfilter-owner@vger.kernel.org> to=<seentr@at.encryp.ch> proto=ESMTP helo=<out1.vger.email>
> | └ update evt.Line.Src : -> /tmp/explain.txt
> | └ update evt.Line.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-05-26 09:57:46.110462548 +0000 UTC
> | └ create evt.Line.Labels.type : syslog
> | └ update evt.Line.Process : %!s(bool=false) -> true
> | └ update evt.Line.Module : -> file
> | └ create evt.Parsed.facility :
> | └ create evt.Parsed.logsource : syslog
> | └ create evt.Parsed.message : 8A96916E0059: milter-reject: END-OF-MESSAGE from out1.vger.email[2620:137:e000::1:20]: 5.7.1 mail 1GXasG from 2620:137:e000::1:20 rejected by DCC; from=<netfilter-owner@vger.kernel.org> to=<seentr@at.encryp.ch> proto=ESMTP helo=<out1.vger.email>
> | └ create evt.Parsed.pid : 2346
> | └ create evt.Parsed.priority :
> | └ create evt.Parsed.program : postfix/cleanup
> | └ create evt.Parsed.timestamp : May 25 10:45:54
> | └ create evt.Parsed.timestamp8601 :
> | └ update evt.Time : 0001-01-01 00:00:00 +0000 UTC -> 2023-05-26 09:57:46.110673438 +0000 UTC
> | └ update evt.StrTime : -> May 25 10:45:54
> | └ create evt.Meta.datasource_path : /tmp/explain.txt
> | └ create evt.Meta.datasource_type : file
> | └ create evt.Meta.machine : flopster
> ├ s01-parse
> | ├ 🔴 crowdsecurity/dovecot-logs
> | ├ 🔴 crowdsecurity/endlessh-logs
> | ├ 🔴 crowdsecurity/nginx-logs
> | ├ 🔴 crowdsecurity/postfix-logs
> | ├ 🔴 crowdsecurity/postscreen-logs
> | ├ 🔴 crowdsecurity/sshd-logs
> | ├ 🔴 g00g1/nginx-custom-logs
> | ├ 🔴 g00g1/postfix-logs-generic
> | ├ 🔴 g00g1/postfix-logs-milter
> | └ 🔴 g00g1/postfix-logs-smtpd
> └-------- parser failure 🔴
> 
> ```

> **g00g1/postfix-logs-milter**
>
> ```auto
> onsuccess: next_stage
> debug: true
> filter: "evt.Parsed.program == 'postfix/cleanup'"
> name: g00g1/postfix-logs-milter
> pattern_syntax:
> POSTFIX_QUEUEID: '([0-9A-F]{6,}|[0-9a-zA-Z]{12,}|NOQUEUE)' # https://github.com/cjslack/grok-debugger/blob/master/public/patterns/postfix
> description: "Parse postfix logs (milter verdict only)"
> nodes:
> - grok:
> apply_on: evt.Parsed.message
> pattern: '%{POSTFIX_QUEUEID:queue_id}: milter-reject: END-OF-MESSAGE from %{POSTFIX_HOSTNAME:remote_host}\[%{IP:remote_addr}\]: 5.\d.\d mail [0-9a-zA-Z]+ from %{IP} rejected by DCC; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=ESMTP helo=<%{HOSTNAME:helo}>'
> statics:
> - meta: log_type_enh
> value: g00g1-dcc-attempt
> statics:
> - meta: service
> value: postfix
> - meta: queue_id
> expression: "evt.Parsed.queue_id"
> - meta: source_ip
> expression: "evt.Parsed.remote_addr"
> - meta: source_hostname
> expression: "evt.Parsed.remote_host"
> - meta: smtp_helo
> expression: "evt.Parsed.helo"
> - meta: smtp_from
> expression: "evt.Parsed.from"
> - meta: smtp_to
> expression: "evt.Parsed.to"
> - meta: log_type
> value: postfix
> 
> ```

I have tested pattern `%{POSTFIX_QUEUEID:queue_id}: milter-reject: END-OF-MESSAGE from %{POSTFIX_HOSTNAME:remote_host}\[%{IP:remote_addr}\]: 5.\d.\d mail [0-9a-zA-Z]+ from %{IP} rejected by DCC; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=ESMTP helo=<%{HOSTNAME:helo}>` and payload `8A96916E0059: milter-reject: END-OF-MESSAGE from out1.vger.email[2620:137:e000::1:20]: 5.7.1 mail 1GXasG from 2620:137:e000::1:20 rejected by DCC; from=<netfilter-owner@vger.kernel.org> to=<seentr@at.encryp.ch> proto=ESMTP helo=<out1.vger.email>` using [grokdebugger.com](http://grokdebugger.com) and everything is parsed as I thought it should be. But crowdsec does not for some reason, and at this point I am unable to do further troubleshooting due to absence of more information in the documentation.

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [May 26, 2023, 10:50am UTC](https://discourse.crowdsec.net/t/issue-with-own-parser/1306/2 "2023-05-26T10:50:41Z")

</div>

The apply\_on just needs to be the key so it should just need to be

```auto
nodes:
  - grok:
      apply_on: message
      pattern: '%{POSTFIX_QUEUEID:queue_id}: milter-reject: END-OF-MESSAGE from %{POSTFIX_HOSTNAME:remote_host}\[%{IP:remote_addr}\]: 5.\d.\d mail [0-9a-zA-Z]+ from %{IP} rejected by DCC; from=<%{EMAILADDRESS:from}> to=<%{EMAILADDRESS:to}> proto=ESMTP helo=<%{HOSTNAME:helo}>'
      statics:
        - meta: log_type_enh
          value: g00g1-dcc-attempt

```

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [May 26, 2023, 11:07am UTC](https://discourse.crowdsec.net/t/issue-with-own-parser/1306/3 "2023-05-26T11:07:07Z")

</div>

Just note this is not very clear in the docs that apply\_on takes a key that is from evt.Parsed object. Will make it clearer

> <https://github.com/crowdsecurity/crowdsec-docs/pull/395>