/usr/bin/ld: cannot find -ldl

crowdsec
22

thanks…

root@ultra:~# go version
go version go1.16.5 linux/arm64

I have applied your modifications and I am testing… (will report results)

This link https://github.com/pyinstaller/pyinstaller/issues/6006#issuecomment-880067770 and all the issue I report on pyinstaller may be of interest !?

OpenWrt does not support on-target compilation. You can’t link the system zlib due to OpenWrt using sstrip which agressively strips symbol table information from shared objects which makes them unsuitable for linking.

This isn’t really a bug that can be solved but rather a consequence of a deliberate design decision (agressivly strip binaries to save space).

23 
root@ultra:~/crowdsec# make release --debug

> root@ultra:~/crowdsec# ls cmd/crowdsec/crowdsec -al
> -rwxr-xr-x    1 root     root      23367680 Jul 16 15:42 cmd/crowdsec/crowdsec
> root@ultra:~/crowdsec# file cmd/crowdsec/crowdsec
> cmd/crowdsec/crowdsec: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=n3ViOIGDeE1bpzkObWf7/NXSzb07aDEeaQADdnNkZ/l8RlVzD1Q0xqBik8zjzr/T7zWXtyUPXfMFFoG-P_X, stripped
> root@ultra:~/crowdsec# file cmd/crowdsec-cli/cscli 
> cmd/crowdsec-cli/cscli: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=wzFlYtazQ4da3eZY3qlo/FGgMc3He9Ykk42B7JVV1/oCs061N2WRskrwnqtGV1/nglqaQcO66Ph0uaRQ_Te, stripped

root@ultra:~/crowdsec# GOOS=linux GOARCH=arm64 make release --debug

> github.com/crowdsecurity/crowdsec/cmd/crowdsec
> # github.com/crowdsecurity/crowdsec/cmd/crowdsec
> /usr/lib/go-1.16/pkg/tool/linux_arm64/link: running gcc failed: exit status 1
> collect2: fatal error: cannot find 'ld'
> compilation terminated.
> 
> make[1]: *** [Makefile:20: build] Error 2
> make: *** [Makefile:90: crowdsec] Error 2
24

Can you try to add export CGO_ENABLED=0 in the Makefile ? (for example, just after the GOARCH ?= amd64 line)

1 Like
25

sure … :wink:

root@ultra:~/crowdsec# GOOS=linux GOARCH=arm64 make release -d
Successfully remade target file 'release'.

root@ultra:~/crowdsec# file cmd/crowdsec/crowdsec
cmd/crowdsec/crowdsec: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=21WlCawwkQEswj_5U3ew/axbwGop5lGYgxOCcGyx3/3kcw10rvEjBYaZrYJxe2/4SsIcU97j2PPHaw6lW-b, stripped
root@ultra:~/crowdsec# file cmd/crowdsec-cli/cscli 
cmd/crowdsec-cli/cscli: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=l3rAMekQ3DOAPQU-Ep6n/UGQy6rBbyaJsoH6vv2Ux/KAeXtB3KHjrfUCSQ__QN/eB2DIU9bG8S8HtZ2lnZk, stripped

It works…!!!

root@ultra:~/crowdsec# cmd/crowdsec/crowdsec --help
Usage of cmd/crowdsec/crowdsec:
  -c string
    	configuration file (default "/etc/crowdsec/config.yaml")
  -debug
    	print debug-level on stdout
  -dsn string
    	Process a single data source in time-machine
  -info
    	print info-level on stdout
  -no-api
    	disable local API
  -no-cs
    	disable crowdsec agent
  -t	only test configs
  -trace
    	VERY verbose
  -type string
    	Labels.type for file in time-machine
  -version
    	display version


root@ultra:~/crowdsec# cmd/crowdsec-cli/cscli --help
cscli is the main command to interact with your crowdsec service, scenarios & db.
It is meant to allow you to manage bans, parsers/scenarios/etc, api and generally manage you crowdsec setup.

Usage:
  cscli [command]

Available Commands:
  alerts        Manage alerts
  bouncers      Manage bouncers [requires local API]
  capi          Manage interaction with Central API (CAPI)
  collections   Manage collections from hub
  completion    Generate completion script
  config        Allows to view current config
  console       Manage interaction with Crowdsec console (https://app.crowdsec.net)
  dashboard     Manage your metabase dashboard container [requires local API]
  decisions     Manage decisions
  help          Help about any command
  hub           Manage Hub
  lapi          Manage interaction with Local API (LAPI)
  machines      Manage local API machines [requires local API]
  metrics       Display crowdsec prometheus metrics.
  parsers       Install/Remove/Upgrade/Inspect parser(s) from hub
  postoverflows Install/Remove/Upgrade/Inspect postoverflow(s) from hub
  scenarios     Install/Remove/Upgrade/Inspect scenario(s) from hub
  simulation    Manage simulation status of scenarios
  version       Display version and exit.

Flags:
  -c, --config string   path to crowdsec config file (default "/etc/crowdsec/config.yaml")
  -o, --output string   Output format : human, json, raw.
      --debug           Set logging to debug.
      --info            Set logging to info.
      --warning         Set logging to warning.
      --error           Set logging to error.
      --trace           Set logging to trace.
  -h, --help            help for cscli

Use "cscli [command] --help" for more information about a command.

and they look to works…

Need now to install (script wizard is missing some commands on OpenWrt !)

THANKS… :wink:

26 
root@ultra:~/crowdsec# ./wizard.sh -i
FATA[07/16/21:16:55:47] crowdsec_wizard: envsubst binary is needed to use do a full install with the wizard, exiting ...

Now that it compiles, we can said that this topic is resolved… will open more about install and testing, and using on OpenWrt specifics !

Edit: not resolved, the go-sqlite3 give errors when the CGO_ENABLED=0 switch is used for crowdsec !

27

looks that it no more use system gcc with this flags… but go/compile !

30

root@ultra:~/go-sqlite3# CGO_ENABLED=1 GOOS=linux GOARCH=arm64 go build github.com/mattn/go-sqlite3

root@ultra:~/go-sqlite3# cscli -c /etc/crowdsec/config.yaml machines add auto
FATA[17-07-2021 07:48:12 AM] unable to create new database client: failed creating schema resources: Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub
31 
root@ultra:~/crowdsec/docker# opkg install yq
Installing yq (4.9.7-1) to root...
Downloading https://downloads.openwrt.org/releases/21.02.0-rc3/packages/aarch64_cortex-a53/packages/yq_4.9.7-1_aarch64_cortex-a53.ipk
Configuring yq.

# install
root@ultra:~# opkg install coreutils-install
Installing coreutils-install (8.32-6) to root...
Downloading https://downloads.openwrt.org/releases/21.02.0-rc3/packages/aarch64_cortex-a53/packages/coreutils-install_8.32-6_aarch64_cortex-a53.ipk
Installing libacl (2.2.53-1) to root...
Downloading https://downloads.openwrt.org/releases/21.02.0-rc3/packages/aarch64_cortex-a53/packages/libacl_2.2.53-1_aarch64_cortex-a53.ipk
Configuring libacl.
Configuring coreutils-install.

# envsubst
root@ultra:~git clone https://github.com/a8m/envsubst.git
root@ultra:~# cd envsubst/cmd/envsubst
root@ultra:~/envsubst/cmd/envsubst# go build -o envsubst main.go
root@ultra:~/envsubst/cmd/envsubst# cp envsubst /usr/bin/

root@ultra:~# envsubst
Usage: envsubst [options...] <input>
Options:
  -i         Specify file input, otherwise use last argument as input file.
             If no input file is specified, read from stdin.
  -o         Specify file output. If none is specified, write to stdout.
  -no-unset  Fail if a variable is not set.
  -no-empty  Fail if a variable is set but empty.
  -fail-fast Fail on first error otherwise display all failures if restrictions are set.
32 
root@ultra:~/crowdsec# ./wizard.sh --docker-mode --force
INFO[07/16/21:20:36:55] crowdsec_wizard: checking existing crowdsec install
WARN[07/16/21:20:36:55] crowdsec_wizard: Crowdsec is already installed !

We recommand to upgrade : sudo ./wizard.sh --upgrade 
If you want to install it anyway, please use '--force'.

Run : sudo ./wizard.sh -i --force
INFO[07/16/21:20:36:55] crowdsec_wizard: installing crowdsec
ln: /usr/bin/cscli: File exists
ln: /usr/bin/crowdsec: File exists

Useful links to start with Crowdsec:

  - Documentation : https://docs.crowdsec.net/Crowdsec/v1/getting_started/crowdsec-tour/
  - Crowdsec Hub  : https://hub.crowdsec.net/ 
  - Open issues   : https://github.com/crowdsecurity/crowdsec/issues

Useful commands to start with Crowdsec:

  - sudo cscli metrics             : https://docs.crowdsec.net/Crowdsec/v1/cscli/cscli_metrics/
  - sudo cscli decisions list      : https://docs.crowdsec.net/Crowdsec/v1/cscli/cscli_decisions_list/
  - sudo cscli alerts list         : https://docs.crowdsec.net/Crowdsec/v1/cscli/cscli_alerts_list/
  - sudo cscli hub list            : https://docs.crowdsec.net/Crowdsec/v1/cscli/cscli_hub_list/
33 
root@ultra:~/crowdsec# ./wizard.sh -i --force
INFO[07/16/21:20:34:00] crowdsec_wizard: checking if crowdsec is installed
WARN[07/16/21:20:34:00] crowdsec_wizard: Crowdsec is already installed !

We recommand to upgrade : sudo ./wizard.sh --upgrade 
If you want to install it anyway, please use '--force'.

Run : sudo ./wizard.sh -i --force
INFO[07/16/21:20:34:00] crowdsec_wizard: installing crowdsec
./wizard.sh: line 403: /etc/systemd/system/crowdsec.service: No such file or directory
FATA[07/16/21:20:34:01] crowdsec_wizard: unable to crowdsec systemd file

OpenWrt is a systemv init style !

I a mtrying to follow manual installation from FreeBSD support · Issue #651 · crowdsecurity/crowdsec · GitHub

34

test with static (modified) :

diff --git a/Makefile b/Makefile
index 7625bf9..467965a 100644
--- a/Makefile
+++ b/Makefile
@@ -21,6 +21,7 @@ BUILD_CMD = "build"
 
 GOOS ?= linux
 GOARCH ?= amd64
+export CGO_ENABLED=0
 
 #Golang version info
 GO_MAJOR_VERSION = $(shell go version | cut -c 14- | cut -d' ' -f1 | cut -d'.' -f1)
@@ -47,7 +48,7 @@ export LD_OPTS_STATIC=-ldflags "-s -w -X github.com/crowdsecurity/crowdsec/pkg/c
 -X github.com/crowdsecurity/crowdsec/pkg/cwversion.Codename=$(BUILD_CODENAME)  \
 -X github.com/crowdsecurity/crowdsec/pkg/cwversion.Tag=$(BUILD_TAG) \
 -X github.com/crowdsecurity/crowdsec/pkg/cwversion.GoVersion=$(BUILD_GOVERSION) \
--extldflags '-static'"
+-linkmode external -extldflags '-static'"
 
 RELDIR = crowdsec-$(BUILD_VERSION)

trying :

root@ultra:~/crowdsec# GOOS=linux GOARCH=arm64 make static

will report (soon)
edit(report): KO : do not build

35

go-sqlite3 need cgo…
arm64 need gold binutils with cgo…
So trying this : The LLVM gold plugin — LLVM 13 documentation

36

useless…
wil try to produce a Makefile for an OpenWrt package and crosscompile it…
I will have to include a custom installation out of wizarf.sh, is there any full manual installation details somewhere ?

37

Here is a final (notes) tutorial/howto (will publish more… soon !)

# makeinfo
wget https://raw.githubusercontent.com/openwrt/openwrt/master/tools/missing-macros/src/bin/makeinfo
chmod a+x makeinfo
cp makeinfo /usr/bin/
opkg install perlbase-config perlbase-getopt

# bison >> https://www.gnu.org/software/bison/
root@ultra:~# 
wget http://ftp.gnu.org/gnu/bison/bison-3.7.6.tar.gz
tar -zxvf bison-3.7.6.tar.gz
cd bison-3.7.6/
./configure --prefix=/usr/
make
make install

# autom4te >> https://www.gnu.org/software/m4/
wget http://ftp.gnu.org/gnu/m4/m4-1.4.19.tar.gz
tar -zxvf m4-1.4.19.tar.gz
cd m4-1.4.19
./configure --prefix=/usr/
make
make install

# binutils >> https://www.gnu.org/software/binutils/ 
# ld.gold >> http://llvm.org/docs/GoldPlugin.html >> git clone --depth 1 git://sourceware.org/git/binutils-gdb.git binutils
wget https://ftp.gnu.org/gnu/binutils/binutils-2.36.tar.gz
tar -zxvf binutils-2.36.tar.gz
cd binutils-2.36
./configure --enable-gold --enable-plugins --disable-werror --prefix=/usr/
make all-gold

cp gold/ld-new /usr/bin/ld.gold

root@ultra:~/binutils-2.36# ldd.gold --version
GNU gold (GNU Binutils 2.36) 1.16
Copyright (C) 2021 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

opkg install yq

cd crowdsec
CGO_ENABLED=1 GOOS=linux GOARCH=arm64 make release --debug

cp crowdsec-v1.1.1/cmd/crowdsec-cli/cscli /usr/bin/cscli 
cp crowdsec-v1.1.1/cmd/crowdsec/crowdsec /usr/bin/crowdsec

root@ultra:~/crowdsec# cscli version
2021/07/18 16:16:19 version: v1.1.1-1610255cbdbb901435d399f8fd533d7267a24adf
2021/07/18 16:16:19 Codename: alphaga
2021/07/18 16:16:19 BuildDate: 2021-07-18_18:14:25
2021/07/18 16:16:19 GoVersion: 1.16.5
2021/07/18 16:16:19 Constraint_parser: >= 1.0, <= 2.0
2021/07/18 16:16:19 Constraint_scenario: >= 1.0, < 3.0
2021/07/18 16:16:19 Constraint_api: v1
2021/07/18 16:16:19 Constraint_acquis: >= 1.0, < 2.0

root@ultra:~/crowdsec# crowdsec --version
2021/07/18 16:16:43 version: v1.1.1-1610255cbdbb901435d399f8fd533d7267a24adf
2021/07/18 16:16:43 Codename: alphaga
2021/07/18 16:16:43 BuildDate: 2021-07-18_18:14:06
2021/07/18 16:16:43 GoVersion: 1.16.5
2021/07/18 16:16:43 Constraint_parser: >= 1.0, <= 2.0
2021/07/18 16:16:43 Constraint_scenario: >= 1.0, < 3.0
2021/07/18 16:16:43 Constraint_api: v1
2021/07/18 16:16:43 Constraint_acquis: >= 1.0, < 2.0

root@ultra:~/crowdsec# cscli -c /etc/crowdsec/config.yaml hub update
INFO[18-07-2021 04:17:25 PM] Wrote new 125878 bytes index to /etc/crowdsec/hub/.index.json

root@ultra:~/crowdsec# mkdir -p /var/lib/crowdsec/data

root@ultra:~/crowdsec# cscli -c /etc/crowdsec/config.yaml machines add --auto
INFO[18-07-2021 04:18:20 PM] Machine 'fca1436b56b347be82c8dd64cbfc2ebaqKlnhEJxIJL285eG' successfully added to the local API 
INFO[18-07-2021 04:18:20 PM] API credentials dumped to '/etc/crowdsec/local_api_credentials.yaml'

root@ultra:~/crowdsec# cscli -c /etc/crowdsec/config.yaml machines list
-----------------------------------------------------------------------------------------------------
 NAME                                              IP ADDRESS  LAST UPDATE           STATUS  VERSION 
-----------------------------------------------------------------------------------------------------
 fca1436b56b347be82c8dd64cbfc2ebaqKlnhEJxIJL285eG              2021-07-18T16:18:20Z  ✔️               
-----------------------------------------------------------------------------------------------------

root@ultra:~/crowdsec# cscli -c /etc/crowdsec/config.yaml capi register
INFO[18-07-2021 04:19:18 PM] Successfully registered to Central API (CAPI) 
INFO[18-07-2021 04:19:18 PM] Central API credentials dumped to '/etc/crowdsec/online_api_credentials.yaml' 
WARN[18-07-2021 04:19:18 PM] Run 'sudo systemctl reload crowdsec' for the new configuration to be effective.

root@ultra:~/crowdsec# crowdsec -c /etc/crowdsec/config.yaml &

root@ultra:~/crowdsec# ps |grep crowdsec
 9908 root      718m S    crowdsec -c /etc/crowdsec/config.yaml
 9973 root      1228 S    grep crowdsec

root@ultra:~/crowdsec# cscli -c /etc/crowdsec/config.yaml hub list
INFO[18-07-2021 04:28:08 PM] Loaded 20 collecs, 25 parsers, 28 scenarios, 3 post-overflow parsers 
INFO[18-07-2021 04:28:08 PM] PARSERS:                                     
--------------------------------------
 NAME  📦 STATUS  VERSION  LOCAL PATH 
--------------------------------------
--------------------------------------
INFO[18-07-2021 04:28:08 PM] SCENARIOS:                                   
--------------------------------------
 NAME  📦 STATUS  VERSION  LOCAL PATH 
--------------------------------------
--------------------------------------
INFO[18-07-2021 04:28:08 PM] COLLECTIONS:                                 
--------------------------------------
 NAME  📦 STATUS  VERSION  LOCAL PATH 
--------------------------------------
--------------------------------------
INFO[18-07-2021 04:28:08 PM] POSTOVERFLOWS:                               
--------------------------------------
 NAME  📦 STATUS  VERSION  LOCAL PATH 
--------------------------------------
--------------------------------------
38 
git clone https://github.com/crowdsecurity/cs-firewall-bouncer.git
cd cs-firewall-bouncer/
CGO_ENABLED=1 GOOS=linux GOARCH=arm64 make release --debug
cp crowdsec-firewall-bouncer.tgz ..
cd ..
tar xzvf crowdsec-firewall-bouncer.tgz
cd crowdsec-firewall-bouncer-v*/
nano install.sh

root@ultra:/crowdsec-firewall-bouncer-v0.0.13# diff -u install.sh.orig install.sh
--- install.sh.orig	2021-07-18 19:20:01.999781496 +0200
+++ install.sh	2021-07-18 19:17:27.757542500 +0200
@@ -18,6 +18,8 @@
         PKG="yum"
     elif [ -f /etc/debian_version ]; then
         PKG="apt"
+    elif [ -f /etc/openwrt_release ]; then
+        PKG="opkg"
     else
         echo "Distribution is not supported, exiting."
         exit
./install.sh

root@ultra:~/crowdsec-firewall-bouncer-v0.0.13# /usr/local/bin/crowdsec-firewall-bouncer -c /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml &
INFO[0000] crowdsec-firewall-bouncer v0.0.13-cc236a78ee262cf4a34bb953c62cbde3350eb4fb


root@ultra:~/crowdsec-firewall-bouncer-v0.0.13# cscli bouncers list
----------------------------------------------------------------------------------------------------------------------------------------------------
 NAME                          IP ADDRESS  VALID  LAST API PULL         TYPE                       VERSION                                          
----------------------------------------------------------------------------------------------------------------------------------------------------
 testBouncer                               ✔️      2021-07-18T17:07:55Z                                                                              
 cs-firewall-bouncer-7PqmCPvp              ✔️      2021-07-18T17:17:37Z                                                                              
 cs-firewall-bouncer-EmuywWTs  127.0.0.1   ✔️      2021-07-18T17:32:48Z  crowdsec-firewall-bouncer  v0.0.13-cc236a78ee262cf4a34bb953c62cbde3350eb4fb 
----------------------------------------------------------------------------------------------------------------------------------------------------
39 
EDIT /etc/crowdsec/config.yaml

  data_dir: /var/lib/crowdsec/data/
>> change to
 data_dir: /srv/crowdsec/data/

mkdir -p /srv/crowdsec/data/ 

cscli -c /etc/crowdsec/config.yaml machines add --auto

cscli -c /etc/crowdsec/config.yaml machines list

cscli -c /etc/crowdsec/config.yaml capi register

/usr/local/bin/crowdsec -c /etc/crowdsec/config.yaml &

cscli collections install crowdsecurity/linux
killall crowdsec
/usr/local/bin/crowdsec -c /etc/crowdsec/config.yaml &


cd /root/crowdsec-firewall-bouncer-v0.0.13/
./install.sh

/usr/local/bin/crowdsec-firewall-bouncer -c /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml &

root@ultra:~/crowdsec-firewall-bouncer-v0.0.13# cscli bouncers list                                                                                                                         
----------------------------------------------------------------------------------------------------------------------------------------------------                                        
 NAME                          IP ADDRESS  VALID  LAST API PULL         TYPE                       VERSION                                                                                  
----------------------------------------------------------------------------------------------------------------------------------------------------                                        
 cs-firewall-bouncer-XxJPObNd  127.0.0.1   ✔️      2021-07-19T15:42:39Z  crowdsec-firewall-bouncer  v0.0.13-cc236a78ee262cf4a34bb953c62cbde3350eb4fb                                         
----------------------------------------------------------------------------------------------------------------------------------------------------                                        



root@ultra:~/crowdsec-firewall-bouncer-v0.0.13# iptables -L | grep crow                                                                                                                     
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists src                                                                                             


ipset list
root@ultra:/# free -h                                                                                                                                                                       
              total        used        free      shared  buff/cache   available                                                                                                             
Mem:        1017188      255592      407876       13980      353720      694096                                                                                                             
Swap:        507900           0      507900
40
1 Like
41

fixed in OpenWrt packaging and fixed upstream !
Thanks…