Local API watcher authentication fails

TLATER · January 27, 2025, 5:58pm

Hi, just trying to get started with crowdsec on a single machine on NixOS, hoping to eventually upstream a module for it, but I can’t even get the service to run.

I’ve gotten to the point that my configuration is read and crowdsec appears to try to start, but then fails with:

level=error msg="unable to open GeoLite2-City.mmdb : open /var/lib/crowdsec/data/GeoLite2-City.mmdb: no such file or directory"
level=info msg="push and pull to Central API disabled"
level=info msg="Enabled feature flags: <none>"
level=info msg="Crowdsec v1.6.3-refs/tags/v1.6.3"
level=info msg="Loading prometheus collectors"
level=warning msg="Communication with CrowdSec Central API disabled from configuration file"
level=info msg="push and pull to Central API disabled"
level=info msg="CrowdSec Local API listening on 127.0.0.1:8080"
level=error msg="unable to open GeoLite2-City.mmdb : open /var/lib/crowdsec/data/GeoLite2-City.mmdb: no such file or directory"
level=warning msg="unable to initialize GeoIP: open /var/lib/crowdsec/data/GeoLite2-City.mmdb: no such file or directory"
level=info msg="Loading grok library /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/patterns"
level=info msg="Loading enrich plugins"
level=info msg="Successfully registered enricher 'GeoIpCity'"
level=info msg="Successfully registered enricher 'GeoIpASN'"
level=info msg="Successfully registered enricher 'IpToRange'"
level=info msg="Successfully registered enricher 'reverse_dns'"
level=info msg="Successfully registered enricher 'ParseDate'"
level=info msg="Successfully registered enricher 'UnmarshalJSON'"
level=info msg="Loading parsers from 0 files"
level=info msg="Loaded 0 nodes from 0 stages"
level=info msg="No postoverflow parsers to load"
level=info msg="Loading 0 scenario files"
level=info msg="Loaded 0 scenarios"
level=info msg="loading acquisition file : /nix/store/3yppyw9yjjxn3vb9l1h8dfvaf8hcxhfy-crowdsec-acquisitions.yaml"
level=warning msg="bad user agent 'crowdsec/v1.6.3-refs/tags/v1.6.3-linux' from '127.0.0.1'"
level=info msg="127.0.0.1 - [Mon, 27 Jan 2025 17:42:48 GMT] \"POST /v1/watchers/login HTTP/1.1 401 47.511301ms \"crowdsec/v1.6.3-refs/tags/v1.6.3-linux\" \""
level=info msg="attempt 1 out of 2"
level=warning msg="bad user agent 'crowdsec/v1.6.3-refs/tags/v1.6.3-linux' from '127.0.0.1'"
level=info msg="127.0.0.1 - [Mon, 27 Jan 2025 17:42:48 GMT] \"POST /v1/watchers/login HTTP/1.1 401 46.661005ms \"crowdsec/v1.6.3-refs/tags/v1.6.3-linux\" \""
level=info msg="attempt 2 out of 2"
level=warning msg="bad user agent 'crowdsec/v1.6.3-refs/tags/v1.6.3-linux' from '127.0.0.1'"
level=info msg="127.0.0.1 - [Mon, 27 Jan 2025 17:42:48 GMT] \"POST /v1/watchers/login HTTP/1.1 401 46.466321ms \"crowdsec/v1.6.3-refs/tags/v1.6.3-linux\" \""
level=info msg="max attempts reached for status code 401"
level=fatal msg="unable to start crowdsec routines: authenticate watcher (1e742f809905409aa4016226cfc8036aTbwFnR6RNGQAzz4a): API error: incorrect Username or Password"

This is right after running:

$ cscli machines add --auto --force

Looking at the existing machines, that seems to have worked fine, and the local_credentials.yaml exists and contains a url, login and password, as expected:

$ cscli machines list
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                                              IP Address  Last Update           Status  Version  OS  Auth Type  Last Heartbeat
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 1e742f809905409aa4016226cfc8036aTbwFnR6RNGQAzz4a  127.0.0.1   2025-01-27T17:38:42Z  ✔️               ?   password   ⚠️ -
 1e742f809905409aa4016226cfc8036avypCcUmOgEjHHhek  127.0.0.1   2025-01-27T17:45:52Z  ✔️               ?   password   ⚠️ -

I’m a bit stumped here. How could authentication fail in this case? Is there a setup step I’m missing? Is the NixOS package and it should report a different user agent?

Detailed systemd services and crowdsec config

If the nix store paths confuse anyone, they’re read-only paths containing the default configuration files from the upstream repository - I’m mostly trying to get to the default state before changing any default configuration, albeit while making sure that crowdsec doesn’t download anything at runtime to limit the stuff I need to understand for now:

api:
  client:
    credentials_path: /var/lib/crowdsec/local_credentials.yaml
  cti:
    enabled: false
    key: null
  server:
    console_path: /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/console.yaml
    listen_uri: 127.0.0.1:8080
    online_client:
      credentials_path: null
    profiles_path: /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/profiles.yaml
common:
  daemonize: false
  log_media: stdout
config_paths:
  config_dir: /etc/crowdsec/
  data_dir: /var/lib/crowdsec/data/
  hub_dir: /var/empty/
  index_path: /nix/store/hrmb80c6ybadp0abn0bsnvnv4vqq8scf-crowdsec-index.json # Contains `{}`
  notification_dir: /var/empty/
  pattern_dir: /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/patterns
  plugin_dir: /var/empty/
  simulation_path: /var/lib/crowdsec/config/simulation.yaml
crowdsec_service:
  acquisition_path: /nix/store/3yppyw9yjjxn3vb9l1h8dfvaf8hcxhfy-crowdsec-acquisitions.yaml
cscli:
  prometheus_uri: 127.0.0.1:6060
db_config:
  db_path: /var/lib/crowdsec/data/crowdsec.db
  flush:
    max_age: 7d
    max_items: 5000
  type: sqlite
  use_wal: true
prometheus:
  enabled: true
  level: full
  listen_addr: 127.0.0.1
  listen_port: 6060

# /etc/systemd/system/crowdsec.service
[Unit]
Description=Crowdsec agent
After=syslog.target network.target remote-fs.target nss-lookup.target

[Service]
Type=notify
Environment=LC_ALL=C LANG=C
ExecStartPre=/nix/store/4knv2cq7pk9n09wv5irhj7zw2cqbny1j-crowdsec-1.6.3/bin/crowdsec -c /etc/crowdsec/config.yaml -t -error
ExecStart=/nix/store/4knv2cq7pk9n09wv5irhj7zw2cqbny1j-crowdsec-1.6.3/bin/crowdsec -c /etc/crowdsec/config.yaml
#ExecStartPost=/bin/sleep 0.1
ExecReload=/nix/store/4knv2cq7pk9n09wv5irhj7zw2cqbny1j-crowdsec-1.6.3/bin/crowdsec -c /etc/crowdsec/config.yaml -t -error
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=60

[Install]
WantedBy=multi-user.target

# /nix/store/0n0jla2l2na15m3zy7jznqrvhdyjsfbd-system-units/crowdsec.service.d/overrides.conf
[Unit]
After=crowdsec-setup.service
BindsTo=crowdsec-setup.service

[Service]
Environment="LOCALE_ARCHIVE=/nix/store/yr4m7nrg1p1qbl7fr2n5h04qh3pnzbzh-glibc-locales-2.40-36/lib/locale/locale-archive"
Environment="PATH=/nix/store/6wgd8c9vq93mqxzc7jhkl86mv6qbc360-coreutils-9.5/bin:/nix/store/r99d2m4swgmrv9jvm4l9di40hvanq1aq-findutils-4.10.0/bin:/nix/store/vniy1y5n8g28c55y7788npwc4h09fh7c-gnugrep-3.11/bin:/nix/store/yq39xdwm4z0fhx7dsm8mlpgvcz3vbfg3-gnused-4.9/bin:/nix/store/bl5dgjbbr9y4wpdw6k959mkq4ig0jwyg-systemd-256.10/bin:/nix/store/6wgd8c9vq93mqxzc7jhkl86mv6qbc360-coreutils-9.5/sbin:/nix/store/r99d2m4swgmrv9jvm4l9di40hvanq1aq-findutils-4.10.0/sbin:/nix/store/vniy1y5n8g28c55y7788npwc4h09fh7c-gnugrep-3.11/sbin:/nix/store/yq39xdwm4z0fhx7dsm8mlpgvcz3vbfg3-gnused-4.9/sbin:/nix/store/bl5dgjbbr9y4wpdw6k959mkq4ig0jwyg-systemd-256.10/sbin"
Environment="TZDIR=/nix/store/78mhfhbhfhvx95hjv9hkjx8m0vadynjv-tzdata-2024b/share/zoneinfo"
Group=crowdsec
StateDirectory=crowdsec
SupplementaryGroups=systemd-journal
User=crowdsec

crowdsec-setup.service is a oneshot unit running this script:

set -e

mkdir -p '/var/lib/crowdsec/'{config,}

if [ ! -e '/var/lib/crowdsec/config/simulation.yaml' ]; then
    cp '/nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/share/crowdsec/config/simulation.yaml' '/var/lib/crowdsec/config/simulation.yaml'
fi

if [ ! -e '/var/lib/crowdsec/local_credentials.yaml' ]; then
    /nix/store/iax65y17x89fvh27hf1fmdfyihzab7gk-crowdsec-with-unit/bin/cscli machines add --auto --file '/var/lib/crowdsec/local_credentials.yaml'
fi

iiAmLoz · January 27, 2025, 6:37pm

By default depending on how you are building the binary itself it should replace the useragent version with the flag provided.

Here is the code that rejects if the username is not valid:

github.com/crowdsecurity/crowdsec

pkg/apiserver/middlewares/v1/jwt.go

5a37161df


      
          		err = j.DbClient.UpdateMachineIP(ctx, clientIP, auth.clientMachine.ID)
          		if err != nil {
          			log.Errorf("Failed to update ip address for '%s': %s\n", auth.clientMachine.MachineId, err)
          			return nil, jwt.ErrFailedAuthentication
          		}
          	}
          
          	useragent := strings.Split(c.Request.UserAgent(), "/")
          	if len(useragent) != 2 {
          		log.Warningf("bad user agent '%s' from '%s'", c.Request.UserAgent(), clientIP)
          		return nil, jwt.ErrFailedAuthentication
          	}
          
          	if err := j.DbClient.UpdateMachineVersion(ctx, useragent[1], auth.clientMachine.ID); err != nil {
          		log.Errorf("unable to update machine '%s' version '%s': %s", auth.clientMachine.MachineId, useragent[1], err)
          		log.Errorf("bad user agent from : %s", clientIP)
          
          		return nil, jwt.ErrFailedAuthentication
          	}
          
          	return &models.WatcherAuthRequest{

So when you build the binary it should build with the BUILD_VERSION= flag

If you dont want to use the Makefile others have got around this by manually setting LD opts

TLATER · January 27, 2025, 7:21pm

Thanks! So the error message is misleading and the actual issue is the user agent rather than user/pass?

I’m using the upstream package, which sets the exact same ldflags. I assume this means that never worked and I should figure out what’s going wrong with the upstream package?

As an aside, the package from that other repository is a literal copy of the upstream one, save for putting the patterns directory in a different location. Neither of those things are necessary, and the repo has many other issues, which is why I’m starting from scratch to actually produce something upstreamable…

iiAmLoz · January 27, 2025, 7:30pm

hmm okay, we set them via this package which gets version and system flag

github.com/crowdsecurity/crowdsec

pkg/apiclient/useragent/useragent.go

5a37161df


      
          package useragent
          
          import (
          	"github.com/crowdsecurity/go-cs-lib/version"
          )
          
          func Default() string {
          	return "crowdsec/" + version.String() + "-" + version.System
          }

TLATER · January 28, 2025, 1:35pm

So I’ve hard-coded the version returned (in pkg/cwversion/version.go because stable NixOS currently packages 1.6.3):

sed '/return "crowdsec/c\return "crowdsec/v${old.version}-linux"' -i pkg/cwversion/version.go

This makes crowdsec not error on login (though the systemd service hangs, I assume turning off daemonize also makes crowdsec not send the startup notification on the systemd socket).

Clearly, the package is broken, presumably the ldflag isn’t being propagated properly. I wonder why other users haven’t noticed; unfortunately the sanity check as written won’t actually do anything to prevent this, since v1.6.3-refs/tags/v1.6.3 matches that grep query.

My best guess is that something in the nixpkgs go build tooling changed, and the handful of users who use crowdsec so far haven’t updated since. Might be a relatively recent regression; time to bisect!

Thanks for the support, and sorry about asking the wrong people, I’d naively assumed the package would be working.

Topic		Replies	Views
Api server init - unable to run local API authenticate watcher	2	276	January 24, 2025
Crowdsec with journald remote/upload crowdsec	6	114	November 1, 2024
CrowdSec error "authenticate watcher" CentOS 8	4	2484	September 24, 2021
Crowdsec Stopped Working on my Raspberry Pi (ARM?) crowdsec	5	955	February 6, 2023
Help Needed: CrowdSec Configuration Error	1	74	July 29, 2024

Local API watcher authentication fails

Related topics