# Local API watcher authentication fails

**URL:** <https://discourse.crowdsec.net/t/local-api-watcher-authentication-fails/2313>\
**Category:** crowdsec\
**Created:** [January 27, 2025, 5:58pm UTC](https://discourse.crowdsec.net/t/local-api-watcher-authentication-fails/2313 "2025-01-27T17:58:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![TLATER](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/tlater/32/1122_2.png) [@TLATER](https://discourse.crowdsec.net/u/TLATER)\
**Post date:** [January 27, 2025, 5:58pm UTC](https://discourse.crowdsec.net/t/local-api-watcher-authentication-fails/2313/1 "2025-01-27T17:58:26Z")

</div>

Hi, just trying to get started with crowdsec on a single machine on NixOS, hoping to eventually upstream a module for it, but I can’t even get the service to run.

I’ve gotten to the point that my configuration is read and crowdsec appears to try to start, but then fails with:

```console
level=error msg="unable to open GeoLite2-City.mmdb : open /var/lib/crowdsec/data/GeoLite2-City.mmdb: no such file or directory"
level=info msg="push and pull to Central API disabled"
level=info msg="Enabled feature flags: <none>"
level=info msg="Crowdsec v1.6.3-refs/tags/v1.6.3"
level=info msg="Loading prometheus collectors"
level=warning msg="Communication with CrowdSec Central API disabled from configuration file"
level=info msg="push and pull to Central API disabled"
level=info msg="CrowdSec Local API listening on 127.0.0.1:8080"
level=error msg="unable to open GeoLite2-City.mmdb : open /var/lib/crowdsec/data/GeoLite2-City.mmdb: no such file or directory"
level=warning msg="unable to initialize GeoIP: open /var/lib/crowdsec/data/GeoLite2-City.mmdb: no such file or directory"
level=info msg="Loading grok library /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/patterns"
level=info msg="Loading enrich plugins"
level=info msg="Successfully registered enricher 'GeoIpCity'"
level=info msg="Successfully registered enricher 'GeoIpASN'"
level=info msg="Successfully registered enricher 'IpToRange'"
level=info msg="Successfully registered enricher 'reverse_dns'"
level=info msg="Successfully registered enricher 'ParseDate'"
level=info msg="Successfully registered enricher 'UnmarshalJSON'"
level=info msg="Loading parsers from 0 files"
level=info msg="Loaded 0 nodes from 0 stages"
level=info msg="No postoverflow parsers to load"
level=info msg="Loading 0 scenario files"
level=info msg="Loaded 0 scenarios"
level=info msg="loading acquisition file : /nix/store/3yppyw9yjjxn3vb9l1h8dfvaf8hcxhfy-crowdsec-acquisitions.yaml"
level=warning msg="bad user agent 'crowdsec/v1.6.3-refs/tags/v1.6.3-linux' from '127.0.0.1'"
level=info msg="127.0.0.1 - [Mon, 27 Jan 2025 17:42:48 GMT] \"POST /v1/watchers/login HTTP/1.1 401 47.511301ms \"crowdsec/v1.6.3-refs/tags/v1.6.3-linux\" \""
level=info msg="attempt 1 out of 2"
level=warning msg="bad user agent 'crowdsec/v1.6.3-refs/tags/v1.6.3-linux' from '127.0.0.1'"
level=info msg="127.0.0.1 - [Mon, 27 Jan 2025 17:42:48 GMT] \"POST /v1/watchers/login HTTP/1.1 401 46.661005ms \"crowdsec/v1.6.3-refs/tags/v1.6.3-linux\" \""
level=info msg="attempt 2 out of 2"
level=warning msg="bad user agent 'crowdsec/v1.6.3-refs/tags/v1.6.3-linux' from '127.0.0.1'"
level=info msg="127.0.0.1 - [Mon, 27 Jan 2025 17:42:48 GMT] \"POST /v1/watchers/login HTTP/1.1 401 46.466321ms \"crowdsec/v1.6.3-refs/tags/v1.6.3-linux\" \""
level=info msg="max attempts reached for status code 401"
level=fatal msg="unable to start crowdsec routines: authenticate watcher (1e742f809905409aa4016226cfc8036aTbwFnR6RNGQAzz4a): API error: incorrect Username or Password"

```

This is right after running:

```console
$ cscli machines add --auto --force

```

Looking at the existing machines, that seems to have worked fine, and the `local_credentials.yaml` exists and contains a url, login and password, as expected:

```console
$ cscli machines list
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name IP Address Last Update Status Version OS Auth Type Last Heartbeat
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 1e742f809905409aa4016226cfc8036aTbwFnR6RNGQAzz4a 127.0.0.1 2025-01-27T17:38:42Z ✔️ ? password ⚠️ -
 1e742f809905409aa4016226cfc8036avypCcUmOgEjHHhek 127.0.0.1 2025-01-27T17:45:52Z ✔️ ? password ⚠️ -

```

I’m a bit stumped here. How could authentication fail in this case? Is there a setup step I’m missing? Is the NixOS package and it should report a different user agent?

> **Detailed systemd services and crowdsec config**
>
> If the nix store paths confuse anyone, they’re read-only paths containing the default configuration files from the upstream repository - I’m mostly trying to get to the default state before changing any default configuration, albeit while making sure that crowdsec doesn’t download anything at runtime to limit the stuff I need to understand for now:
> 
> ```yaml
> api:
> client:
> credentials_path: /var/lib/crowdsec/local_credentials.yaml
> cti:
> enabled: false
> key: null
> server:
> console_path: /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/console.yaml
> listen_uri: 127.0.0.1:8080
> online_client:
> credentials_path: null
> profiles_path: /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/profiles.yaml
> common:
> daemonize: false
> log_media: stdout
> config_paths:
> config_dir: /etc/crowdsec/
> data_dir: /var/lib/crowdsec/data/
> hub_dir: /var/empty/
> index_path: /nix/store/hrmb80c6ybadp0abn0bsnvnv4vqq8scf-crowdsec-index.json # Contains `{}`
> notification_dir: /var/empty/
> pattern_dir: /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/patterns
> plugin_dir: /var/empty/
> simulation_path: /var/lib/crowdsec/config/simulation.yaml
> crowdsec_service:
> acquisition_path: /nix/store/3yppyw9yjjxn3vb9l1h8dfvaf8hcxhfy-crowdsec-acquisitions.yaml
> cscli:
> prometheus_uri: 127.0.0.1:6060
> db_config:
> db_path: /var/lib/crowdsec/data/crowdsec.db
> flush:
> max_age: 7d
> max_items: 5000
> type: sqlite
> use_wal: true
> prometheus:
> enabled: true
> level: full
> listen_addr: 127.0.0.1
> listen_port: 6060
> 
> ```
> 
> ```systemd
> # /etc/systemd/system/crowdsec.service
> [Unit]
> Description=Crowdsec agent
> After=syslog.target network.target remote-fs.target nss-lookup.target
> 
> [Service]
> Type=notify
> Environment=LC_ALL=C LANG=C
> ExecStartPre=/nix/store/4knv2cq7pk9n09wv5irhj7zw2cqbny1j-crowdsec-1.6.3/bin/crowdsec -c /etc/crowdsec/config.yaml -t -error
> ExecStart=/nix/store/4knv2cq7pk9n09wv5irhj7zw2cqbny1j-crowdsec-1.6.3/bin/crowdsec -c /etc/crowdsec/config.yaml
> #ExecStartPost=/bin/sleep 0.1
> ExecReload=/nix/store/4knv2cq7pk9n09wv5irhj7zw2cqbny1j-crowdsec-1.6.3/bin/crowdsec -c /etc/crowdsec/config.yaml -t -error
> ExecReload=/bin/kill -HUP $MAINPID
> Restart=always
> RestartSec=60
> 
> [Install]
> WantedBy=multi-user.target
> 
> ```
> 
> ```systemd
> # /nix/store/0n0jla2l2na15m3zy7jznqrvhdyjsfbd-system-units/crowdsec.service.d/overrides.conf
> [Unit]
> After=crowdsec-setup.service
> BindsTo=crowdsec-setup.service
> 
> [Service]
> Environment="LOCALE_ARCHIVE=/nix/store/yr4m7nrg1p1qbl7fr2n5h04qh3pnzbzh-glibc-locales-2.40-36/lib/locale/locale-archive"
> Environment="PATH=/nix/store/6wgd8c9vq93mqxzc7jhkl86mv6qbc360-coreutils-9.5/bin:/nix/store/r99d2m4swgmrv9jvm4l9di40hvanq1aq-findutils-4.10.0/bin:/nix/store/vniy1y5n8g28c55y7788npwc4h09fh7c-gnugrep-3.11/bin:/nix/store/yq39xdwm4z0fhx7dsm8mlpgvcz3vbfg3-gnused-4.9/bin:/nix/store/bl5dgjbbr9y4wpdw6k959mkq4ig0jwyg-systemd-256.10/bin:/nix/store/6wgd8c9vq93mqxzc7jhkl86mv6qbc360-coreutils-9.5/sbin:/nix/store/r99d2m4swgmrv9jvm4l9di40hvanq1aq-findutils-4.10.0/sbin:/nix/store/vniy1y5n8g28c55y7788npwc4h09fh7c-gnugrep-3.11/sbin:/nix/store/yq39xdwm4z0fhx7dsm8mlpgvcz3vbfg3-gnused-4.9/sbin:/nix/store/bl5dgjbbr9y4wpdw6k959mkq4ig0jwyg-systemd-256.10/sbin"
> Environment="TZDIR=/nix/store/78mhfhbhfhvx95hjv9hkjx8m0vadynjv-tzdata-2024b/share/zoneinfo"
> Group=crowdsec
> StateDirectory=crowdsec
> SupplementaryGroups=systemd-journal
> User=crowdsec
> 
> ```
> 
> `crowdsec-setup.service` is a oneshot unit running this script:
> 
> ```bash
> set -e
> 
> mkdir -p '/var/lib/crowdsec/'{config,}
> 
> if [! -e '/var/lib/crowdsec/config/simulation.yaml']; then
> cp '/nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/simulation.yaml' '/var/lib/crowdsec/config/simulation.yaml'
> fi
> 
> if [! -e '/var/lib/crowdsec/local_credentials.yaml']; then
> /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/bin/cscli machines add --auto --file '/var/lib/crowdsec/local_credentials.yaml'
> fi
> 
> ```

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [January 27, 2025, 6:37pm UTC](https://discourse.crowdsec.net/t/local-api-watcher-authentication-fails/2313/2 "2025-01-27T18:37:35Z")

</div>

By default depending on how you are building the binary itself it should replace the useragent version with the flag provided.

Here is the code that rejects if the username is not valid:

> <https://github.com/crowdsecurity/crowdsec/blob/5a37161df268ee4ded05b5a4c9f3adb138c8eeed/pkg/apiserver/middlewares/v1/jwt.go#L236>

So when you build the binary it should build with the `BUILD_VERSION=` flag

If you dont want to use the Makefile others have got around this by manually setting LD opts

> **[nix-flake-crowdsec/packages/crowdsec/default.nix at...](https://codeberg.org/kampka/nix-flake-crowdsec/src/commit/c6aa259c883e3810167b754fed72fc06119734a0/packages/crowdsec/default.nix#L58)**
>
> nix-flake-crowdsec - A nix flake for running Crowdsec on NixOS

---

<div class="post-metadata">

**Author:** ![TLATER](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/tlater/32/1122_2.png) [@TLATER](https://discourse.crowdsec.net/u/TLATER)\
**Post date:** [January 27, 2025, 7:21pm UTC](https://discourse.crowdsec.net/t/local-api-watcher-authentication-fails/2313/3 "2025-01-27T19:21:22Z")

</div>

Thanks! So the error message is misleading and the actual issue is the user agent rather than user/pass?

I’m using the upstream package, which sets the [exact same ldflags](https://github.com/NixOS/nixpkgs/blob/nixos-24.11/pkgs/by-name/cr/crowdsec/package.nix#L31). I assume this means that never worked and I should figure out what’s going wrong with the upstream package?

As an aside, the package from that other repository is a literal copy of the upstream one, save for putting the `patterns` directory in a different location. Neither of those things are necessary, and the repo has many other issues, which is why I’m starting from scratch to actually produce something upstreamable…

---

<div class="post-metadata">

**Author:** ![iiAmLoz](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/iiamloz/32/386_2.png) [@iiAmLoz](https://discourse.crowdsec.net/u/iiAmLoz)\
**Post date:** [January 27, 2025, 7:30pm UTC](https://discourse.crowdsec.net/t/local-api-watcher-authentication-fails/2313/4 "2025-01-27T19:30:23Z")

</div>

hmm okay, we set them via this package which gets version and system flag

> <https://github.com/crowdsecurity/crowdsec/blob/5a37161df268ee4ded05b5a4c9f3adb138c8eeed/pkg/apiclient/useragent/useragent.go#L1-L9>

---

<div class="post-metadata">

**Author:** ![TLATER](https://dub1.discourse-cdn.com/flex013/user_avatar/discourse.crowdsec.net/tlater/32/1122_2.png) [@TLATER](https://discourse.crowdsec.net/u/TLATER)\
**Post date:** [January 28, 2025, 1:35pm UTC](https://discourse.crowdsec.net/t/local-api-watcher-authentication-fails/2313/5 "2025-01-28T13:35:38Z")

</div>

So I’ve hard-coded the version returned (in `pkg/cwversion/version.go` because stable NixOS currently packages 1.6.3):

```auto
sed '/return "crowdsec/c\return "crowdsec/v${old.version}-linux"' -i pkg/cwversion/version.go

```

This makes crowdsec not error on login (though the systemd service hangs, I assume turning off `daemonize` _also_ makes crowdsec not send the startup notification on the systemd socket).

Clearly, the package is broken, presumably the ldflag isn’t being propagated properly. I wonder why other users haven’t noticed; unfortunately [the sanity check](https://github.com/NixOS/nixpkgs/blob/nixos-24.11/pkgs/by-name/cr/crowdsec/package.nix#L53) as written won’t actually do anything to prevent this, since `v1.6.3-refs/tags/v1.6.3` matches that `grep` query.

My best guess is that something in the nixpkgs go build tooling changed, and the handful of users who use crowdsec so far haven’t updated since. Might be a relatively recent regression; time to bisect!

Thanks for the support, and sorry about asking the wrong people, I’d naively assumed the package would be working.