# Crowdsec with journald remote/upload

**URL:** <https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113>\
**Category:** crowdsec\
**Created:** [October 31, 2024, 8:34pm UTC](https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113 "2024-10-31T20:34:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![munnik](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/munnik/32/1027_2.png) [@munnik](https://discourse.crowdsec.net/u/munnik)\
**Post date:** [October 31, 2024, 8:34pm UTC](https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113/1 "2024-10-31T20:34:09Z")

</div>

I’m running Crowdsec on Nixos and I’m having troubles to block http crawlers. Crowdsec is running on the host machine with the following nix config, using this [kampka/nix-flake-crowdsec: A nix flake for running Crowdsec on NixOS - Codeberg.org](https://codeberg.org/kampka/nix-flake-crowdsec)

```auto
{
  inputs,
  config,
  pkgs,
  lib,
  ...
}:
let
  api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx";
  ifTheyExist = groups: builtins.filter (group: builtins.hasAttr group config.users.groups) groups;
in
{
  imports = [
    inputs.crowdsec.nixosModules.crowdsec
    inputs.crowdsec.nixosModules.crowdsec-firewall-bouncer
  ];

  sops.secrets = {
    crowdsec-enroll-key = {
      sopsFile = ./secrets.yaml;
      owner = "crowdsec";
    };
  };
  services.crowdsec = {
    enable = true;
    allowLocalJournalAccess = true;
    acquisitions =
      let
        transports = [
          "journal"
          "syslog"
          "stdout"
          "kernel"
        ];
      in
      map (transport: {
        source = "journalctl";
        journalctl_filter = ["_TRANSPORT=${transport}"];
        labels.type = "syslog";
      }) transports
      ++ map (transport: {
        source = "journalctl";
        journalctl_filter = [
          "_TRANSPORT=${transport}"
          "--directory=/var/log/journal/remote"
        ];
        labels.type = "syslog";
      }) transports;
    enrollKeyFile = config.sops.secrets.crowdsec-enroll-key.path;
    settings = {
      api.server = {
        listen_uri = "localhost:8080";
      };
    };
  };
  nixpkgs.overlays = [inputs.crowdsec.overlays.default];
  services.crowdsec-firewall-bouncer = {
    enable = true;
    settings = {
      api_key = api_key;
      api_url = "http://localhost:8080";
      deny_log = true;
      iptables_chains = [
        "INPUT"
        "FORWARD"
      ];
    };
  };
  systemd.services.crowdsec.serviceConfig = {
    ExecStartPre =
      let
        crowdsecInstalls = {
          bouncers = ["crowdsec-firewall-bouncer"];
          collections = [
            "crowdsecurity/base-http-scenarios"
            "crowdsecurity/http-dos"
            "crowdsecurity/iptables"
            "crowdsecurity/linux"
            "crowdsecurity/nginx"
            "crowdsecurity/sshd"
            "crowdsecurity/whitelist-good-actors"
            "crowdsecurity/wireguard"
          ];
        };
        script = pkgs.writeScriptBin "setup-crowdsec" (
          ''
            #!${pkgs.runtimeShell}
            set -eu
            set -o pipefail
          ''
          + lib.strings.concatStrings (
            lib.lists.flatten (
              lib.mapAttrsToList (
                name: installs:
                lib.forEach installs (install: ''
                  if ! cscli ${name} list --output raw | grep -q "${install}"; then
                    cscli ${name} install ${install}            
                  fi
                '')
              ) crowdsecInstalls
            )
          )
        );
      in
      ["${script}/bin/setup-crowdsec"];
  };
  users.users.crowdsec.extraGroups = ifTheyExist ["systemd-journal-remote"];
}

```

nginx is running in a Nixos container and its log messages are passed using journald remote/upload. That is why I’ve setup the `journalctl_filter` to include `--directory=/var/log/journal/remote`. And I added an extra group `systemd-journal-remote` to the `crowdsec` user. I _think_ crowdsec is reading the logs from the nginx container.

```auto
$ sudo cscli metrics
Acquisition Metrics:
╭──────────────────────────────────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────╮
│ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├──────────────────────────────────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ : │ - │ - │ - │ - │ 382 │
│ journalctl:journalctl-_TRANSPORT=journal │ 174 │ - │ 174 │ - │ - │
│ journalctl:journalctl-_TRANSPORT=journal.--directory=/var/log/journal/remote │ 2 │ - │ 2 │ - │ - │
│ journalctl:journalctl-_TRANSPORT=kernel │ 1.86k │ 1.86k │ - │ 805 │ - │
│ journalctl:journalctl-_TRANSPORT=stdout │ 189.09k │ - │ 189.09k │ - │ - │
│ journalctl:journalctl-_TRANSPORT=syslog │ 343 │ 67 │ 276 │ 202 │ - │
│ journalctl:journalctl-_TRANSPORT=syslog.--directory=/var/log/journal/remote │ 121.79k │ 121.74k │ 57 │ 209.37k │ - │
╰──────────────────────────────────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯

Local API Alerts:
╭────────────────────────────────────────────┬───────╮
│ Reason │ Count │
├────────────────────────────────────────────┼───────┤
│ crowdsecurity/ssh-bf │ 28 │
│ crowdsecurity/ssh-slow-bf │ 43 │
│ crowdsecurity/ssh-slow-bf_user-enum │ 1 │
│ crowdsecurity/CVE-2022-41082 │ 2 │
│ crowdsecurity/http-admin-interface-probing │ 2 │
│ crowdsecurity/http-cve-2021-41773 │ 45 │
│ crowdsecurity/http-cve-2021-42013 │ 28 │
│ crowdsecurity/iptables-scan-multi_ports │ 2135 │
│ crowdsecurity/thinkphp-cve-2018-20062 │ 137 │
│ manual 'ban' from 'hetzner-otap01' │ 1 │
│ crowdsecurity/http-bad-user-agent │ 48 │
│ crowdsecurity/http-open-proxy │ 14 │
│ crowdsecurity/netgear_rce │ 2 │
│ crowdsecurity/CVE-2019-18935 │ 1 │
│ crowdsecurity/http-dos-swithcing-ua │ 6 │
│ crowdsecurity/http-wordpress-scan │ 1 │
│ crowdsecurity/ssh-bf_user-enum │ 2 │
│ ltsich/http-w00tw00t │ 2 │
│ crowdsecurity/CVE-2017-9841 │ 32 │
│ crowdsecurity/http-crawl-non_statics │ 2 │
│ crowdsecurity/http-probing │ 54 │
│ crowdsecurity/http-sensitive-files │ 8 │
│ crowdsecurity/jira_cve-2021-26086 │ 14 │
╰────────────────────────────────────────────┴───────╯

Bouncer Metrics (crowdsec-firewall-bouncer) since 2024-10-11 23:18:57 +0000 UTC:
╭──────────────────────────────────┬──────────────────┬───────────────────┬───────────────────────╮
│ Origin │ active_decisions │ dropped │ processed │
│ │ IPs │ bytes │ packets │ bytes │ packets │
├──────────────────────────────────┼──────────────────┼─────────┼─────────┼───────────┼───────────┤
│ CAPI (community blocklist) │ 55.58k │ 11.41M │ 252.61k │ - │ - │
│ crowdsec (security engine) │ 1 │ 1.16M │ 24.63k │ - │ - │
│ cscli (manual decisions) │ 0 │ 840 │ 14 │ - │ - │
│ lists:firehol_cruzit_web_attacks │ 13.10k │ 440 │ 10 │ - │ - │
│ lists:firehol_greensnow │ 1.98k │ 21.48k │ 392 │ - │ - │
│ lists:otx-webscanners │ 7.19k │ 158.34k │ 3.50k │ - │ - │
├──────────────────────────────────┼──────────────────┼─────────┼─────────┼───────────┼───────────┤
│ Total │ 77.86k │ 12.75M │ 281.16k │ 3.87T │ 2.48G │
╰──────────────────────────────────┴──────────────────┴─────────┴─────────┴───────────┴───────────╯

Local API Decisions:
╭──────────────────────────────────────────────┬──────────┬────────┬───────╮
│ Reason │ Origin │ Action │ Count │
├──────────────────────────────────────────────┼──────────┼────────┼───────┤
│ crowdsecurity/CVE-2023-49103 │ CAPI │ ban │ 21 │
│ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI │ ban │ 44 │
│ crowdsecurity/ssh-bf │ CAPI │ ban │ 7592 │
│ crowdsecurity/ssh-bf │ crowdsec │ ban │ 1 │
│ crowdsecurity/CVE-2017-9841 │ CAPI │ ban │ 483 │
│ crowdsecurity/CVE-2022-37042 │ CAPI │ ban │ 2 │
│ crowdsecurity/fortinet-cve-2018-13379 │ CAPI │ ban │ 17 │
│ crowdsecurity/nginx-req-limit-exceeded │ CAPI │ ban │ 641 │
│ crowdsecurity/f5-big-ip-cve-2020-5902 │ CAPI │ ban │ 2 │
│ crowdsecurity/http-backdoors-attempts │ CAPI │ ban │ 223 │
│ crowdsecurity/http-dos-invalid-http-versions │ CAPI │ ban │ 1508 │
│ crowdsecurity/http-open-proxy │ CAPI │ ban │ 2419 │
│ crowdsecurity/netgear_rce │ CAPI │ ban │ 170 │
│ crowdsecurity/thinkphp-cve-2018-20062 │ CAPI │ ban │ 318 │
│ crowdsecurity/CVE-2022-26134 │ CAPI │ ban │ 8 │
│ crowdsecurity/http-wordpress-scan │ CAPI │ ban │ 1315 │
│ firehol_cruzit_web_attacks │ lists │ ban │ 13252 │
│ crowdsecurity/http-crawl-non_statics │ CAPI │ ban │ 583 │
│ crowdsecurity/http-cve-probing │ CAPI │ ban │ 62 │
│ crowdsecurity/http-dos-random-uri │ CAPI │ ban │ 1 │
│ crowdsecurity/http-path-traversal-probing │ CAPI │ ban │ 274 │
│ crowdsecurity/CVE-2019-18935 │ CAPI │ ban │ 136 │
│ crowdsecurity/CVE-2022-44877 │ CAPI │ ban │ 1 │
│ crowdsecurity/CVE-2023-22515 │ CAPI │ ban │ 3 │
│ crowdsecurity/http-admin-interface-probing │ CAPI │ ban │ 270 │
│ crowdsecurity/http-bad-user-agent │ CAPI │ ban │ 19696 │
│ crowdsecurity/http-cve-2021-41773 │ CAPI │ ban │ 791 │
│ crowdsecurity/http-generic-bf │ CAPI │ ban │ 42 │
│ crowdsecurity/http-probing │ CAPI │ ban │ 7906 │
│ crowdsecurity/http-sensitive-files │ CAPI │ ban │ 523 │
│ crowdsecurity/jira_cve-2021-26086 │ CAPI │ ban │ 22 │
│ otx-webscanners │ lists │ ban │ 8487 │
│ crowdsecurity/iptables-scan-multi_ports │ CAPI │ ban │ 911 │
│ crowdsecurity/spring4shell_cve-2022-22965 │ CAPI │ ban │ 1 │
│ crowdsecurity/ssh-slow-bf │ CAPI │ ban │ 9595 │
│ ltsich/http-w00tw00t │ CAPI │ ban │ 3 │
│ firehol_greensnow │ lists │ ban │ 7309 │
│ crowdsecurity/CVE-2022-35914 │ CAPI │ ban │ 7 │
│ crowdsecurity/http-cve-2021-42013 │ CAPI │ ban │ 9 │
│ crowdsecurity/ssh-cve-2024-6387 │ CAPI │ ban │ 57 │
╰──────────────────────────────────────────────┴──────────┴────────┴───────╯

Local API Metrics:
╭──────────────────────┬────────┬──────╮
│ Route │ Method │ Hits │
├──────────────────────┼────────┼──────┤
│ /v1/alerts │ GET │ 9 │
│ /v1/decisions/stream │ GET │ 385 │
│ /v1/heartbeat │ GET │ 71 │
│ /v1/usage-metrics │ POST │ 18 │
│ /v1/watchers/login │ POST │ 18 │
╰──────────────────────┴────────┴──────╯

Local API Bouncers Metrics:
╭───────────────────────────┬──────────────────────┬────────┬──────╮
│ Bouncer │ Route │ Method │ Hits │
├───────────────────────────┼──────────────────────┼────────┼──────┤
│ crowdsec-firewall-bouncer │ /v1/decisions/stream │ GET │ 385 │
╰───────────────────────────┴──────────────────────┴────────┴──────╯

Local API Machines Metrics:
╭────────────────┬───────────────┬────────┬──────╮
│ Machine │ Route │ Method │ Hits │
├────────────────┼───────────────┼────────┼──────┤
│ hetzner-otap01 │ /v1/alerts │ GET │ 9 │
│ hetzner-otap01 │ /v1/heartbeat │ GET │ 71 │
╰────────────────┴───────────────┴────────┴──────╯

Parser Metrics:
╭──────────────────────────────────┬─────────┬─────────┬──────────╮
│ Parsers │ Hits │ Parsed │ Unparsed │
├──────────────────────────────────┼─────────┼─────────┼──────────┤
│ child-crowdsecurity/http-logs │ 365.21k │ 299.97k │ 65.24k │
│ child-crowdsecurity/nginx-logs │ 182.43k │ 121.74k │ 60.70k │
│ child-crowdsecurity/sshd-logs │ 717 │ 67 │ 650 │
│ child-crowdsecurity/syslog-logs │ 313.27k │ 313.24k │ 26 │
│ crowdsecurity/cdn-whitelist │ 382 │ 382 │ - │
│ crowdsecurity/dateparse-enrich │ 123.66k │ 123.66k │ - │
│ crowdsecurity/geoip-enrich │ 123.66k │ 123.66k │ - │
│ crowdsecurity/http-logs │ 121.74k │ 121.74k │ - │
│ crowdsecurity/iptables-logs │ 1.86k │ 1.86k │ - │
│ crowdsecurity/nginx-logs │ 121.78k │ 121.74k │ 44 │
│ crowdsecurity/rdns │ 382 │ 382 │ - │
│ crowdsecurity/seo-bots-whitelist │ 382 │ 382 │ - │
│ crowdsecurity/sshd-logs │ 98 │ 67 │ 31 │
│ crowdsecurity/syslog-logs │ 313.26k │ 313.24k │ 13 │
╰──────────────────────────────────┴─────────┴─────────┴──────────╯

Scenario Metrics:
╭────────────────────────────────────────────┬───────────────┬───────────┬──────────────┬─────────┬─────────╮
│ Scenario │ Current Count │ Overflows │ Instantiated │ Poured │ Expired │
├────────────────────────────────────────────┼───────────────┼───────────┼──────────────┼─────────┼─────────┤
│ LePresidente/http-generic-401-bf │ - │ - │ 3 │ 3 │ 3 │
│ crowdsecurity/http-admin-interface-probing │ - │ - │ 2 │ 2 │ 2 │
│ crowdsecurity/http-bad-user-agent │ 41 │ 326 │ 1.37k │ 1.82k │ 999 │
│ crowdsecurity/http-crawl-non_statics │ 501 │ - │ 56.69k │ 104.90k │ 56.19k │
│ crowdsecurity/http-dos-swithcing-ua │ 570 │ - │ 20.35k │ 48.81k │ 19.78k │
│ crowdsecurity/http-probing │ 679 │ 164 │ 16.56k │ 53.84k │ 15.72k │
│ crowdsecurity/http-sensitive-files │ - │ - │ 1 │ 1 │ 1 │
│ crowdsecurity/iptables-scan-multi_ports │ 12 │ - │ 671 │ 805 │ 659 │
│ crowdsecurity/ssh-bf │ 1 │ - │ 34 │ 67 │ 33 │
│ crowdsecurity/ssh-bf_user-enum │ 1 │ - │ 34 │ 34 │ 33 │
│ crowdsecurity/ssh-slow-bf │ 4 │ - │ 9 │ 67 │ 5 │
│ crowdsecurity/ssh-slow-bf_user-enum │ 4 │ - │ 9 │ 34 │ 5 │
╰────────────────────────────────────────────┴───────────────┴───────────┴──────────────┴─────────┴─────────╯

Whitelist Metrics:
╭──────────────────────────────────┬────────────────────────────────────┬──────┬─────────────╮
│ Whitelist │ Reason │ Hits │ Whitelisted │
├──────────────────────────────────┼────────────────────────────────────┼──────┼─────────────┤
│ crowdsecurity/cdn-whitelist │ CDN provider │ 382 │ 382 │
│ crowdsecurity/seo-bots-whitelist │ good bots (search engine crawlers) │ 382 │ - │
╰──────────────────────────────────┴────────────────────────────────────┴──────┴─────────────╯

```

The output of the explain command makes my think that the filters are working correctly.

```auto
$ cscli explain --dsn "journalctl://filters=_TRANSPORT=syslog&filters=--directory=/var/log/journal/remote&filters=-n 5" --type syslog
line: Oct 31 21:28:59 nginx nginx[417]: nginx nginx: 172.70.100.188 - - [31/Oct/2024:21:28:59 +0100] "GET /u/5c0ac03fbd9eee6fb21393f3 HTTP/1.1" 404 180 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.69 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
	├ s00-raw
	|	└ 🟢 crowdsecurity/syslog-logs (+11 ~7)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	└ 🟢 crowdsecurity/nginx-logs (+22 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	└ 🟢 crowdsecurity/http-logs (+7)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		├ 🟢 crowdsecurity/http-dos-swithcing-ua
		└ 🟢 crowdsecurity/http-probing

line: Oct 31 21:28:59 nginx nginx[417]: nginx nginx: 2024/10/31 21:28:59 [error] 417#417: *223768 open() "/nix/store/1499nwmqw6ksiwgv2iq5b0rf4aq5wgqc-website/blablabla" failed (2: No such file or directory), client: 85.10.135.177, server: hetzner-otap01.sustainablemotion.io, request: "GET /blablabla HTTP/2.0", host: "hetzner-otap01.sustainablemotion.io"
	├ s00-raw
	|	└ 🟢 crowdsecurity/syslog-logs (+11 ~7)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	└ 🟢 crowdsecurity/nginx-logs (+15 ~3)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	└ 🟢 crowdsecurity/http-logs (+7)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		└ 🟢 crowdsecurity/http-dos-swithcing-ua

line: Oct 31 21:28:59 nginx nginx[417]: nginx nginx: 85.10.135.177 - - [31/Oct/2024:21:28:59 +0100] "GET /blablabla HTTP/2.0" 404 146 "-" "curl/7.74.0"
	├ s00-raw
	|	└ 🟢 crowdsecurity/syslog-logs (+11 ~7)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	└ 🟢 crowdsecurity/nginx-logs (+22 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	└ 🟢 crowdsecurity/http-logs (+7)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		├ 🟢 crowdsecurity/http-dos-swithcing-ua
		└ 🟢 crowdsecurity/http-probing

line: Oct 31 21:28:59 nginx nginx[417]: nginx nginx: 2024/10/31 21:28:59 [error] 417#417: *222581 open() "/nix/store/1499nwmqw6ksiwgv2iq5b0rf4aq5wgqc-website/a/5ebadf0186ec4d3d4b290ae3" failed (2: No such file or directory), client: 162.158.203.49, server: hetzner-otap01.sustainablemotion.io, request: "GET /a/5ebadf0186ec4d3d4b290ae3?lang=zh-cn&amp=1 HTTP/1.1", host: "www.xuehua.us"
	├ s00-raw
	|	└ 🟢 crowdsecurity/syslog-logs (+11 ~7)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	└ 🟢 crowdsecurity/nginx-logs (+15 ~3)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	└ 🟢 crowdsecurity/http-logs (+8 ~1)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		└ 🟢 crowdsecurity/http-dos-swithcing-ua

line: Oct 31 21:28:59 nginx nginx[417]: nginx nginx: 162.158.203.49 - - [31/Oct/2024:21:28:59 +0100] "GET /a/5ebadf0186ec4d3d4b290ae3?lang=zh-cn&amp=1 HTTP/1.1" 404 180 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6723.69 Mobile Safari/537.36 (compatible; GoogleOther)"
	├ s00-raw
	|	└ 🟢 crowdsecurity/syslog-logs (+11 ~7)
	├ s01-parse
	|	├ 🔴 crowdsecurity/iptables-logs
	|	└ 🟢 crowdsecurity/nginx-logs (+22 ~2)
	├ s02-enrich
	|	├ 🟢 crowdsecurity/dateparse-enrich (+2 ~1)
	|	├ 🟢 crowdsecurity/geoip-enrich (+13)
	|	└ 🟢 crowdsecurity/http-logs (+8 ~1)
	├-------- parser success 🟢
	├ Scenarios
		├ 🟢 crowdsecurity/http-crawl-non_statics
		├ 🟢 crowdsecurity/http-dos-swithcing-ua
		└ 🟢 crowdsecurity/http-probing

```

Problem is that the nginx logs are full of `404` request but I get very few `http-probing` alerts and no decisions:

```auto
cscli decisions list
╭─────────┬──────────┬────────────────────┬──────────────────────┬────────┬─────────┬─────────────────────┬────────┬────────────┬──────────╮
│ ID │ Source │ Scope:Value │ Reason │ Action │ Country │ AS │ Events │ expiration │ Alert ID │
├─────────┼──────────┼────────────────────┼──────────────────────┼────────┼─────────┼─────────────────────┼────────┼────────────┼──────────┤
│ 4454401 │ crowdsec │ Ip:103.174.130.251 │ crowdsecurity/ssh-bf │ ban │ IN │ 147287 DATAPARADISE │ 6 │ 49m26s │ 2938 │
╰─────────┴──────────┴────────────────────┴──────────────────────┴────────┴─────────┴─────────────────────┴────────┴────────────┴──────────╯

```

On another machine I ran (also see the output of the `cscli explain` command):

```auto
watch -n 0.3 curl -k -s -f https://hetzner-otap01.sustainablemotion.io/blablabla

```

But it doesn’t appear in the alerts or decisions.

What should I change/test in my configuration to make this working?

My config:

```auto
$ cscli config show
Global:
   - Configuration Folder : /var/lib/crowdsec/config
   - Data Folder : /var/lib/crowdsec/data
   - Hub Folder : /var/lib/crowdsec/hub
   - Simulation File : /nix/store/whadgb8bh7mplg2vbk2hj0a4bz76130k-crowdsec-1.6.3/share/crowdsec/config/simulation.yaml
   - Log Folder :
   - Log level : <nil>
   - Log Media : stdout
Crowdsec:
  - Acquisition File :
  - Parsers routines : 0
  - Acquisition Folder : /nix/store/hc3738lcdk4g5pfq59pcnrsiay6d94qp-crowdsec-acquisitions
cscli:
  - Output : human
  - Hub Branch :
API Client:
  - URL : http://127.0.0.1:8080/
  - Login : hetzner-otap01
  - Credentials File : /var/lib/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL : localhost:8080
  - Listen Socket :
  - Profile File : /nix/store/whadgb8bh7mplg2vbk2hj0a4bz76130k-crowdsec-1.6.3/share/crowdsec/config/profiles.yaml

  - Trusted IPs:
  - Database:
      - Type : sqlite
      - Path : /var/lib/crowdsec/data/crowdsec.db

```

---

<div class="post-metadata">

**Author:** ![munnik](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/munnik/32/1027_2.png) [@munnik](https://discourse.crowdsec.net/u/munnik)\
**Post date:** [October 31, 2024, 9:17pm UTC](https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113/2 "2024-10-31T21:17:41Z")

</div>

I removed `crowdsecurity/whitelist-good-actors` and now hosts are blocked!

```auto
$ cscli decisions list
╭─────────┬──────────┬────────────────────┬───────────────────────────────────────┬────────┬─────────┬──────────────────────────────────┬────────┬────────────┬──────────╮
│ ID │ Source │ Scope:Value │ Reason │ Action │ Country │ AS │ Events │ expiration │ Alert ID │
├─────────┼──────────┼────────────────────┼───────────────────────────────────────┼────────┼─────────┼──────────────────────────────────┼────────┼────────────┼──────────┤
│ 4512030 │ crowdsec │ Ip:162.158.202.136 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 25 │ 3h59m42s │ 2967 │
│ 4512029 │ crowdsec │ Ip:172.71.103.216 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h59m40s │ 2966 │
│ 4512028 │ crowdsec │ Ip:172.71.102.185 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h59m30s │ 2965 │
│ 4512027 │ crowdsec │ Ip:162.158.202.132 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 32 │ 3h59m18s │ 2964 │
│ 4512026 │ crowdsec │ Ip:172.70.47.164 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h59m9s │ 2963 │
│ 4512025 │ crowdsec │ Ip:162.158.202.154 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 18 │ 3h59m7s │ 2962 │
│ 4512024 │ crowdsec │ Ip:172.70.46.28 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h59m6s │ 2961 │
│ 4512023 │ crowdsec │ Ip:162.158.202.146 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 20 │ 3h59m4s │ 2960 │
│ 4512022 │ crowdsec │ Ip:162.158.202.128 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 27 │ 3h58m35s │ 2959 │
│ 4512021 │ crowdsec │ Ip:172.71.183.41 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h58m21s │ 2958 │
│ 4512020 │ crowdsec │ Ip:162.158.202.125 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 25 │ 3h58m17s │ 2957 │
│ 4512019 │ crowdsec │ Ip:172.70.47.107 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h58m12s │ 2956 │
│ 4512018 │ crowdsec │ Ip:172.71.182.123 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h58m5s │ 2955 │
│ 4512017 │ crowdsec │ Ip:162.158.202.144 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 24 │ 3h57m58s │ 2954 │
│ 4512016 │ crowdsec │ Ip:162.158.202.150 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 23 │ 3h57m53s │ 2953 │
│ 4512015 │ crowdsec │ Ip:162.158.202.153 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 23 │ 3h57m43s │ 2952 │
│ 4512014 │ crowdsec │ Ip:162.158.202.155 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 20 │ 3h57m27s │ 2951 │
│ 4512013 │ crowdsec │ Ip:162.158.202.126 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 20 │ 3h57m16s │ 2950 │
│ 4512012 │ crowdsec │ Ip:20.14.108.37 │ crowdsecurity/thinkphp-cve-2018-20062 │ ban │ US │ 8075 MICROSOFT-CORP-MSN-AS-BLOCK │ 1 │ 3h57m12s │ 2949 │
│ 4512011 │ crowdsec │ Ip:162.158.202.137 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 16 │ 3h56m38s │ 2948 │
│ 4512010 │ crowdsec │ Ip:162.158.202.140 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 16 │ 3h56m35s │ 2947 │
│ 4512009 │ crowdsec │ Ip:162.158.202.147 │ crowdsecurity/http-probing │ ban │ DE │ 13335 CLOUDFLARENET │ 15 │ 3h56m31s │ 2946 │
│ 4512008 │ crowdsec │ Ip:172.71.182.192 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h56m7s │ 2945 │
│ 4512007 │ crowdsec │ Ip:172.71.99.56 │ crowdsecurity/http-bad-user-agent │ ban │ NL │ 13335 CLOUDFLARENET │ 2 │ 3h56m2s │ 2944 │
╰─────────┴──────────┴────────────────────┴───────────────────────────────────────┴────────┴─────────┴──────────────────────────────────┴────────┴────────────┴──────────╯

```

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [November 1, 2024, 9:24am UTC](https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113/3 "2024-11-01T09:24:40Z")

</div>

> I removed `crowdsecurity/whitelist-good-actors` and now hosts are blocked!

If this is the case your nginx is not configured correctly to get the IP address from cloudflare proxied headers, as currently you are banning the Cloudflare IP address so that means all users that go through cloudflare (which should be all) are all being banned so nobody can get to your website anymore. (this depends if the user closet cloudflare proxy is banned but you need to resolve this as you want the real ip of the user)

> **[Restoring original visitor IPs | Cloudflare Support docs](https://developers.cloudflare.com/support/troubleshooting/restoring-visitor-ips/restoring-original-visitor-ips/#nginx-1)**
>
> When your website traffic is routed through the Cloudflare network ↗, we act as a reverse proxy. This allows Cloudflare to speed up page load time by routing packets more efficiently and caching static resources (images, JavaScript, CSS, etc.)....

I also wrote a quick script that can automate this, but for nixos I dont know if this is right approach

> <https://gist.github.com/LaurenceJJones/7e7e386bb1f0dae7054c95df2722a4bb>

---

<div class="post-metadata">

**Author:** ![munnik](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/munnik/32/1027_2.png) [@munnik](https://discourse.crowdsec.net/u/munnik)\
**Post date:** [November 1, 2024, 9:54am UTC](https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113/4 "2024-11-01T09:54:48Z")

</div>

Thank you for your reply. We are not using Cloudflare:

```auto
$ host -t any hetzner-otap01.sustainablemotion.io
hetzner-otap01.sustainablemotion.io has address 95.216.78.164
hetzner-otap01.sustainablemotion.io has RRSIG record A 13 3 3600 20241114000000 20241024000000 18902 sustainablemotion.io. iy3w8e0wyaPGYfWCaEpWWL4SpK/rJI1ibml83L7XxKz2JCRxY0PekbxL jmT4n9YcRe1Ue2qiI1GWd84bkLtduw==

```

So, as I understand, the attacker is using Cloudflare to access our website? How should I block these attacks because the firewall only knows the IP address?

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [November 1, 2024, 10:36am UTC](https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113/5 "2024-11-01T10:36:02Z")

</div>

Very odd, I would contact hetzner to understand how and why Cloudflare CDN range is hitting your webserver then as Cloudflare ranges shouldnt be hitting your server at all if you are not using them.

Unless you inherited somebodies IP address that has forgotten to remove their A records? Might be useful to prepend the hostname to the nginx logs so you can see if they are requesting a resource on your server that doesnt exists.

example nginx config:

```auto
    log_format main '$server_name $remote_addr - $remote_user [$time_local] '
                '"$request" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent"';
    # Logging
    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log warn;

```

but I know the access\_log for you might be different

even thought the hostname might not show as $server\_name as depending on your nginx configuration it could be hitting the last in chain.

Edit: I just did so OSINT on the ip address I can see that a domain did have your IP address before so maybe they were using cloudflare and they have forgotten to disable their A record that is being proxied by cloudflare OR this hetzner instance is being used by multiple people but this would just be very odd as you are getting the logs for all requests so this is probably not the case.

Edit edit: Just did a lookup on their domain and they are using cloudflare, I dont want to leak the domain but are you pointing two different domains towards this IP?

---

<div class="post-metadata">

**Author:** ![munnik](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/munnik/32/1027_2.png) [@munnik](https://discourse.crowdsec.net/u/munnik)\
**Post date:** [November 1, 2024, 11:42am UTC](https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113/6 "2024-11-01T11:42:54Z")

</div>

Thank you again, this was really helpful. I found the other hostname pointing to our IP address. I contacted Hetzner, and hopefully they can help me further. After that, I will re-enable the whitelist.

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [November 1, 2024, 11:55am UTC](https://discourse.crowdsec.net/t/crowdsec-with-journald-remote-upload/2113/7 "2024-11-01T11:55:46Z")

</div>

Well, hetzner might not be able to help and since you dont use Cloudflare so this is _okay_ for now for you to block those IP’s since you cannot control this and it definitely should be resolved by the user that owns the other domain, I tried to see if I could find a contact address but it seems like all of theirs sites are gone.