# NGiNX Logs in GELF JSON Format

**URL:** <https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53>\
**Category:** crowdsec\
**Created:** [November 3, 2020, 12:01am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53 "2020-11-03T00:01:30Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![grant-veepshosting](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/grant-veepshosting/32/24_2.png) [@grant-veepshosting](https://discourse.crowdsec.net/u/grant-veepshosting)\
**Post date:** [November 3, 2020, 12:01am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/1 "2020-11-03T00:01:30Z")

</div>

Hi All,

Love the idea of this security solution, and we are actively trialling it in non-production currently.

We are currently hamstrung from contributing our log files to the upstream analysis and analytics, because our access+error logs in NGiNX are in GELF-JSON format, for ingestion into Graylog2.

Has anyone solved this, or a similar problem? We have considered the following approaches -

1. Write a 2nd log for each vHost in the default “combined” format. There doesn’t seem to be any simple way of doing this, however.
2. Write a custom GROK pattern filter to override the default “combined” interpretation. Are there any pre existing filters for the GELF JSON format already? Or guides for newbies writing GROK filters?

Example GELF JSON format logs, from nginx.conf -  
` log_format gelf_json escape=json '{ "timestamp": "$time_iso8601", ' '"remote_addr": "$remote_addr", ' '"connection": "$connection", ' '"connection_requests": $connection_requests, ' '"pipe": "$pipe", ' '"body_bytes_sent": $body_bytes_sent, ' '"request_length": $request_length, ' '"request_time": $request_time, ' '"response_status": $status, ' '"request": "$request", ' '"request_method": "$request_method", ' '"host": "$host", ' '"upstream_cache_status": "$upstream_cache_status", ' '"upstream_addr": "$upstream_addr", ' '"http_x_forwarded_for": "$http_x_forwarded_for", ' '"http_referrer": "$http_referer", ' '"http_user_agent": "$http_user_agent", ' '"http_version": "$server_protocol", ' '"remote_user": "$remote_user", ' '"http_x_forwarded_proto": "$http_x_forwarded_proto", ' '"upstream_response_time": "$upstream_response_time", ' '"nginx_access": true }';`

---

<div class="post-metadata">

**Author:** ![kaa](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/kaa/32/17_2.png) [@kaa](https://discourse.crowdsec.net/u/kaa)\
**Post date:** [November 3, 2020, 8:09am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/2 "2020-11-03T08:09:05Z")

</div>

Hi,

I believe the produced logs are JSON, aren’t they ?

So we have no built-in parser that can be fed with json logs for now, but we have already some features to do it. You’ll find an example in the unit tests [https://github.com/crowdsecurity/crowdsec/tree/master/pkg/parser/tests/base-json-extract](https://github.com/crowdsecurity/crowdsec/tree/master/pkg/parser/tests/base-json-extract)

I guess the resulting parser file should look like this:

```auto
filter: "evt.Parsed.program startsWith 'nginx'"
onsuccess: next_stage
#debug: true
name: crowdsecurity/nginx-logs
description: "Parse nginx access and error logs"
statics:
  - target: evt.StrTime
    expression: JsonExtract(evt.Line.Raw, "timestamp8601")
  - parsed: "logsource"
    value: "gelf-nginx"
  - meta: source_ip
    expression: JsonExtract(evt.Line.Raw, "remote_addr")
  - meta: http_status
    expression: JsonExtract(evt.Line.Raw, "response_status")
  - meta: http_path
    expression: JsonExtract(evt.Line.Raw, "request")
  - meta: log_type
    value: http_access-log

```

This file should be put in the `/etc/crowdsec/config/parsers/s00-raw` directory but I can’t test it because I don’t have any of your logs. If you want to, you can provide us a sample of your logs and we’ll have a better chance to provide you a ready to use parser file.

---

<div class="post-metadata">

**Author:** ![grant-veepshosting](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/grant-veepshosting/32/24_2.png) [@grant-veepshosting](https://discourse.crowdsec.net/u/grant-veepshosting)\
**Post date:** [November 3, 2020, 8:54am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/3 "2020-11-03T08:54:22Z")

</div>

Hi Kaa,

Thanks very much for the prompt and informative reply.

Please find an example log file here: [https://www.veepshosting.com/ssl-redacted.domain.co.access.log.gz](https://www.veepshosting.com/ssl-redacted.domain.co.access.log.gz)

---

<div class="post-metadata">

**Author:** ![grant-veepshosting](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/grant-veepshosting/32/24_2.png) [@grant-veepshosting](https://discourse.crowdsec.net/u/grant-veepshosting)\
**Post date:** [November 4, 2020, 7:10am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/4 "2020-11-04T07:10:35Z")

</div>

I’m getting conflicting feedback on whether or not it’s working.

The primary log file ‘/var/log/crowdsec.log’ has the following entry, using the parser above -  
time=“04-11-2020 18:06:01” level=debug msg="+ Processing 2 statics" func=“[github.com/crowdsecurity/crowdsec/pkg/parser.(\*Node](http://github.com/crowdsecurity/crowdsec/pkg/parser.(*Node)).process” file="/home/runner/work/crowdsec/crowdsec/pkg/parser/node.go:313" id=long-lake name=crowdsecurity/non-syslog stage=s00-raw  
time=“04-11-2020 18:06:01” level=debug msg=".Parsed[message] = ‘{ “timestamp”: “2020-11-04T18:06:01+11:00”, “remote\_addr”: “54.162.224.1”, “connection”: “50343”, “connection\_requests”: 1, “pipe”: “.”, “body\_bytes\_sent”: 161205, “request\_length”: 325, “request\_time”: 0.228, “response\_status”: 200, “request”: “GET /media/catalog/product/b/e/bella\_rosa.jpg HTTP/1.1”, “request\_method”: “GET”, “host”: “[redacted.domain.co](http://redacted.domain.co)”, “upstream\_cache\_status”: “”, “upstream\_addr”: “”, “http\_x\_forwarded\_for”: “”, “http\_referrer”: “”, “http\_user\_agent”: “Ruby”, “http\_version”: “HTTP/1.1”, “remote\_user”: “”, “http\_x\_forwarded\_proto”: “”, “upstream\_response\_time”: “”, “nginx\_access”: true }’" [func=github.com/crowdsecurity/crowdsec/pkg/parser.ProcessStatics](http://func=github.com/crowdsecurity/crowdsec/pkg/parser.ProcessStatics) file="/home/runner/work/crowdsec/crowdsec/pkg/parser/runtime.go:175" id=long-lake name=crowdsecurity/non-syslog stage=s00-raw

Yes the metrics command doesn’t show any lines parsed? -

```
sudo cscli metrics

```

INFO[0000] Buckets Metrics:  
±-------±--------------±----------±-------------±-------±--------+  
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |  
±-------±--------------±----------±-------------±-------±--------+  
±-------±--------------±----------±-------------±-------±--------+  
INFO[0000] Acquisition Metrics:  
±----------------------------------------------------------±-----------±-------------±---------------±-----------------------+  
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |  
±----------------------------------------------------------±-----------±-------------±---------------±-----------------------+  
| /var/log/auth.log | 24 | - | 24 | - |  
| /var/log/nginx/ssl-obfuscated.domain.co.access.log | 132 | - | 132 | - |  
| /var/log/syslog | 11 | - | 11 | - |  
±----------------------------------------------------------±-----------±-------------±---------------±-----------------------+  
INFO[0000] Parser Metrics:  
±------------------------------±-----±-------±---------+  
| PARSERS | HITS | PARSED | UNPARSED |  
±------------------------------±-----±-------±---------+  
| child-crowdsecurity/sshd-logs | 10 | - | 10 |  
| crowdsecurity/non-syslog | 132 | 132 | - |  
| crowdsecurity/sshd-logs | 2 | - | 2 |  
| crowdsecurity/syslog-logs | 35 | 35 | - |  
±------------------------------±-----±-------±---------+

---

<div class="post-metadata">

**Author:** ![kaa](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/kaa/32/17_2.png) [@kaa](https://discourse.crowdsec.net/u/kaa)\
**Post date:** [November 5, 2020, 11:19am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/5 "2020-11-05T11:19:09Z")

</div>

The nginx-gelp thingy gets only unparsed logs. So this is not working?

OTOH the installation seems functional because syslog did parse 35 log lines. But this parser has to be followed by another one (and nginx-gelp thingy is not eligible). Your configuration seems that beside your nginx-gelp thingy only ssh is enabled thus this one only is able to trigger anything, and it was fed only by two lines of log. This seems actually legit.

I’ll take some time to dig into your logs very soon.

---

<div class="post-metadata">

**Author:** ![kaa](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/kaa/32/17_2.png) [@kaa](https://discourse.crowdsec.net/u/kaa)\
**Post date:** [November 5, 2020, 7:08pm UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/6 "2020-11-05T19:08:02Z")

</div>

Ok I got the parsing to work.

1. You’ll have to configure your `/etc/crowdsec/config/acquis.yaml` with something like

```auto
filenames:
  - /var/log/<the generated gelp-nginx log file>
labels:
  type: gelp-nginx
---

```

1. Then add the folllowing configuration as parsing file to add in `/etc/crowdsec/config/parsers/s00-raw`. Whatever name with the extension .yaml will do.

```auto
filter: "evt.Line.Labels.type == 'nginx-gelp'"
onsuccess: next_stage
#debug: true
name: crowdsecurity/nginx-logs
description: "Parse nginx access and error logs"
statics:
  - target: evt.StrTime
    expression: JsonExtract(evt.Line.Raw, "timestamp")
  - parsed: "logsource"
    value: "gelf-nginx"
  - parsed: remote_addr
    expression: JsonExtract(evt.Line.Raw, "remote_addr")   
  - parsed: remote_user
    expression: JsonExtract(evt.Line.Raw, "remote_user")   
  - meta: source_ip
    expression: JsonExtract(evt.Line.Raw, "remote_addr")
  - meta: http_status
    expression: JsonExtract(evt.Line.Raw, "response_status")
  - meta: http_path
    expression: JsonExtract(evt.Line.Raw, "request")
  - meta: log_type
    value: http_access-log
  - meta: service
    value: http
  - parsed: http_user_agent
    expression: JsonExtract(evt.Line.Raw, "http_user_agent")
  - parsed: http_referer
    expression: JsonExtract(evt.Line.Raw, "http_referrer")
  - parsed: target_fqdn
    expression: JsonExtract(evt.Line.Raw, "host")
  - parsed: method
    expression: JsonExtract(evt.Line.Raw, "request_method")
  - parsed: body_bytes_sent
    expression: JsonExtract(evt.Line.Raw, "body_bytes_sent")
  - parsed: http_version
    expression: JsonExtract(evt.Line.Raw, "http_version")   
  - parsed: status
    expression: JsonExtract(evt.Line.Raw, "response_status")

```

Please keep in mind that the expression `"evt.Line.Labels.type == 'nginx-gelp'" as to match the label in the `acquis.yaml` file.

1. A final step is required to make all the dot connect with the http-related scenarios. Add the following file in `/etc/crowdsec/config/parsers/s01-parse`. Whatever name with `.yaml` extension will do:

```auto
filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']"
onsuccess: next_stage
name: local/gelp-nginx-request
nodes:
  - grok:
      pattern: '%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}'
      apply_on: full_request

```

---

<div class="post-metadata">

**Author:** ![grant-veepshosting](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/grant-veepshosting/32/24_2.png) [@grant-veepshosting](https://discourse.crowdsec.net/u/grant-veepshosting)\
**Post date:** [November 10, 2020, 4:48am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/7 "2020-11-10T04:48:54Z")

</div>

Thanks Kaa, we are testing it currently, however the log files that match are still sitting in “Unparsed”. No errors relating to the above config are listen in the main crowdsec.log log file, so possibly just a matter of waiting a bit longer.

---

<div class="post-metadata">

**Author:** ![grant-veepshosting](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/grant-veepshosting/32/24_2.png) [@grant-veepshosting](https://discourse.crowdsec.net/u/grant-veepshosting)\
**Post date:** [November 10, 2020, 5:56am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/8 "2020-11-10T05:56:09Z")

</div>

cscli metrics  
INFO[0000] Buckets Metrics:  
±-------------------------------±--------------±----------±-------------±-------±--------+  
| BUCKET | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |  
±-------------------------------±--------------±----------±-------------±-------±--------+  
| crowdsecurity/ssh-bf | - | - | 12 | 13 | 12 |  
| crowdsecurity/ssh-bf\_user-enum | - | - | 12 | 12 | 12 |  
±-------------------------------±--------------±----------±-------------±-------±--------+  
INFO[0000] Acquisition Metrics:  
±-------------------------------------------------------------±-----------±-------------±---------------±-----------------------+  
| SOURCE | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |  
±-------------------------------------------------------------±-----------±-------------±---------------±-----------------------+  
| /var/log/auth.log | 877 | 13 | 864 | 25 |  
| /var/log/nginx/ssl-server.access.log | 1 | - | 1 | - |  
| /var/log/nginx/ssl-server.error.log | 1 | - | 1 | - |  
| /var/log/nginx/ssl-preprod.site.co.access.log | 1 | - | 1 | - |  
| /var/log/nginx/ssl-test.site.co.access.log | 1814 | - | 1814 | - |  
| /var/log/syslog | 517 | - | 517 | - |  
±-------------------------------------------------------------±-----------±-------------±---------------±-----------------------+  
INFO[0000] Parser Metrics:  
±-------------------------------±-----±-------±---------+  
| PARSERS | HITS | PARSED | UNPARSED |  
±-------------------------------±-----±-------±---------+  
| child-crowdsecurity/sshd-logs | 836 | 13 | 823 |  
| child-local/gelf-nginx | 526 | - | 526 |  
| crowdsecurity/dateparse-enrich | 13 | 13 | - |  
| crowdsecurity/geoip-enrich | 13 | 13 | - |  
| crowdsecurity/nginx-logs | 526 | 526 | - |  
| crowdsecurity/non-syslog | 1291 | 1291 | - |  
| crowdsecurity/sshd-logs | 171 | 13 | 158 |  
| crowdsecurity/syslog-logs | 1394 | 1394 | - |  
| crowdsecurity/whitelists | 13 | 13 | - |  
| local/gelf-nginx | 526 | - | 526 |  
±-------------------------------±-----±-------±---------+

---

<div class="post-metadata">

**Author:** ![kaa](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/kaa/32/17_2.png) [@kaa](https://discourse.crowdsec.net/u/kaa)\
**Post date:** [November 10, 2020, 7:48am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/9 "2020-11-10T07:48:05Z")

</div>

Hi @grant-veepshosting,

You definitely should get more count in the parsed column.

But when I read the config I pasted you I found a nasty typo: ` type: gelp-nginx` in the 1. should match the `nginx-gelp` in the filter lin in the 2. These two entries should match in order the parser to know which type of log it’s working on.

Replacing you `/etc/crowdsec/config/acquis.yaml` with

```auto
filenames:
  - /var/log/<the generated gelp-nginx log file>
labels:
  type: nginx-gelp

```

should do the trick

Sorry for this

---

<div class="post-metadata">

**Author:** ![grant-veepshosting](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/grant-veepshosting/32/24_2.png) [@grant-veepshosting](https://discourse.crowdsec.net/u/grant-veepshosting)\
**Post date:** [November 10, 2020, 10:06pm UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/10 "2020-11-10T22:06:31Z")

</div>

Hey Kaa,

I noticed this typo and corrected it yesterday, thanks for following up. Also gelp should be gelf.

According to the debug logging, it’s being processed, but not according to any cscli metrics / statistics.

Example debug log entry, anonymised -

[https://www.veepshosting.com/nginx\_gelf\_debug\_example.log.gz](https://www.veepshosting.com/nginx_gelf_debug_example.log.gz)

---

<div class="post-metadata">

**Author:** ![kaa](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/kaa/32/17_2.png) [@kaa](https://discourse.crowdsec.net/u/kaa)\
**Post date:** [November 12, 2020, 11:38am UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/11 "2020-11-12T11:38:12Z")

</div>

hi @grant-veepshosting,  
Yes, indeed. But it stil lacks a step. The ` s00-raw` step is working fine. But, the `s01-parse` step isn’t.  
I guess a typo still got through my testing because the `apply_on` directive on the last file doesn’t match anything coming from the `s00-raw` stage. To fix this you’ll have to add the following line of the file in the `s00-parse` stage:

```auto
  - parsed: full_request
    expression: JsonExtract(evt.Line.Raw, "request")

```

Furthermore, from the last `cscli metrics` output you showed us that the `base-http-scenario` collection is not installed. I wrote the whole gelf (sorry for me being confused with the p over the f) for being compatible with this collection.

You can do `cscli install collection crowdsecurity/base-http-scenarios` to install it.

---

<div class="post-metadata">

**Author:** ![grant-veepshosting](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/grant-veepshosting/32/24_2.png) [@grant-veepshosting](https://discourse.crowdsec.net/u/grant-veepshosting)\
**Post date:** [November 17, 2020, 10:17pm UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/12 "2020-11-17T22:17:21Z")

</div>

Thank you kaa!!! It works not, thanks very much for sticking in there.

Would you like me to post the configs in full for reference, or possible addition to the code base?

PS: I had to add the section above “request” to the s00-raw/_yaml file, not the s01-parse/_.yaml file for it to work correctly.

---

<div class="post-metadata">

**Author:** ![kaa](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/kaa/32/17_2.png) [@kaa](https://discourse.crowdsec.net/u/kaa)\
**Post date:** [November 17, 2020, 10:28pm UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/13 "2020-11-17T22:28:15Z")

</div>

HI @grant-veepshosting,

yes, it would be great to have this. At some point, we may want to add this to the official stuff in the hub.

Thanks for your feedback !

---

<div class="post-metadata">

**Author:** ![grant-veepshosting](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/grant-veepshosting/32/24_2.png) [@grant-veepshosting](https://discourse.crowdsec.net/u/grant-veepshosting)\
**Post date:** [November 17, 2020, 10:29pm UTC](https://discourse.crowdsec.net/t/nginx-logs-in-gelf-json-format/53/14 "2020-11-17T22:29:12Z")

</div>

/etc/crowdsec/config/parsers/s00-raw/nginx-gelf.yaml:  
`filter: "evt.Line.Labels.type == 'gelf-nginx'"`  
`onsuccess: next_stage`  
`debug: true`  
`name: crowdsecurity/nginx-logs`  
`description: "Parse nginx access and error logs"`  
`statics:`  
` - target: evt.StrTime`  
` expression: JsonExtract(evt.Line.Raw, "timestamp")`  
` - parsed: "logsource"`  
` value: "gelf-nginx"`  
` - parsed: remote_addr`  
` expression: JsonExtract(evt.Line.Raw, "remote_addr")`  
` - parsed: remote_user`  
` expression: JsonExtract(evt.Line.Raw, "remote_user")`  
` - meta: source_ip`  
` expression: JsonExtract(evt.Line.Raw, "remote_addr")`  
` - meta: http_status`  
` expression: JsonExtract(evt.Line.Raw, "response_status")`  
` - meta: http_path`  
` expression: JsonExtract(evt.Line.Raw, "request")`  
` - meta: log_type`  
` value: http_access-log`  
` - meta: service`  
` value: http`  
` - parsed: http_user_agent`  
` expression: JsonExtract(evt.Line.Raw, "http_user_agent")`  
` - parsed: http_referer`  
` expression: JsonExtract(evt.Line.Raw, "http_referrer")`  
` - parsed: target_fqdn`  
` expression: JsonExtract(evt.Line.Raw, "host")`  
` - parsed: method`  
` expression: JsonExtract(evt.Line.Raw, "request_method")`  
` - parsed: body_bytes_sent`  
` expression: JsonExtract(evt.Line.Raw, "body_bytes_sent")`  
` - parsed: http_version`  
` expression: JsonExtract(evt.Line.Raw, "http_version")`  
` - parsed: status`  
` expression: JsonExtract(evt.Line.Raw, "response_status")`  
` - parsed: full_request`  
` expression: JsonExtract(evt.Line.Raw, "request")`

/etc/crowdsec/config/parsers/s01-parse/nginx-gelf-logs.yaml:  
`filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']"`  
`onsuccess: next_stage`  
`name: local/gelf-nginx`  
`nodes:`  
` - grok:`  
` pattern: '%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}'`  
` apply_on: full_request`