CAPI and APP/CONSOLE and upgrade v1.1.1 to v1.2.0

Question : If using WebConsole with a registered instance, how can I update/upgrade my Crowdsec instance without any lost of statistics ?
How can I re-register to same instance ID ?

Do I need to remove the CAPI register and do it only if never did ?

cscli -c /etc/crowdsec/config.yaml capi register

Did my local machine-id will be used ?

In the OpenWrt Package, I execute this commands at post-install (first install but also update / upgrade…) :

cscli -c /etc/crowdsec/config.yaml machines add --force "$(cat /var/lib/dbus/machine-id)" -a -f /etc/crowdsec/local_api_credentials.yaml
cscli -c /etc/crowdsec/config.yaml capi register
cscli hub update && cscli collections install crowdsecurity/linux && cscli parsers install crowdsecurity/whitelists

Must (and can) I do it better ?
Thanks in advance…

Hello @Gandalf,

Do I need to remove the CAPI register and do it only if never did ?

Yes, this command must be run only one time (at the installation) and shouldn’t be run when upgrading the package.

If you can don’t run the cscli capi register at upgrade it would be great :slight_smile:

1 Like

Thanks…

I will change my script to :

[ -s /etc/crowdsec/online_api_credentials.yaml ] || cscli -c /etc/crowdsec/config.yaml capi register -f /etc/crowdsec/online_api_credentials.yaml

The command will execute only if the online_api_credentials.yaml is empty, which is the case of first installation !

Yes this should be good, perfect :slight_smile:

1 Like

Done an upgrade to 1.2.0 on my main routeur :
got this now :

root@LPM:~/custom/1.2.0# cscli machines list
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 NAME                                              IP ADDRESS  LAST UPDATE                STATUS  VERSION                                                                
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 5244e15d7908402192135ac72b4acb10yqGbFd9bcN233Mpf  10.4.2.16   2021-09-20T05:26:59+02:00  ✔️       v1.1.1-debian-pragmatic-linux-73e0bbaf93070f4a640eb5a22212b5dcf26699de 
 2f827bc3cfb84cb0b1e59a5ae3f1492dIdIeA8T6UWmvJqhV  10.4.2.159  2021-08-16T15:33:13+02:00  ✔️       v1.1.1-debian-pragmatic-linux-73e0bbaf93070f4a640eb5a22212b5dcf26699de 
 fd738d07d6d54d8ca63b14fba8078980fy4ABpU4hSONMAyO  127.0.0.1   2021-09-20T02:15:01+02:00  ✔️       v1.1.1-openwrt-73e0bbaf93070f4a640eb5a22212b5dcf26699de                
                                                   127.0.0.1   2021-09-20T15:02:45+02:00  ✔️       v1.2.0-openwrt-openwrt                                                 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

oups… bug on OpenWrt 19.07.x not machine-id file !
cat /var/lib/dbus/machine-id (return nothing)…
only correct since 21.02.x !

I did some cleanup (empty id machine and re-register of the new software with same id as before) !
May be this was a mistake to re-register ?

root@LPM:~/custom/1.2.0# cscli machines list
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 NAME                                              IP ADDRESS  LAST UPDATE                STATUS  VERSION                                                                
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 5244e15d7908402192135ac72b4acb10yqGbFd9bcN233Mpf  10.4.2.16   2021-09-20T05:26:59+02:00  ✔️       v1.1.1-debian-pragmatic-linux-73e0bbaf93070f4a640eb5a22212b5dcf26699de 
 2f827bc3cfb84cb0b1e59a5ae3f1492dIdIeA8T6UWmvJqhV  10.4.2.159  2021-08-16T15:33:13+02:00  ✔️       v1.1.1-debian-pragmatic-linux-73e0bbaf93070f4a640eb5a22212b5dcf26699de 
 fd738d07d6d54d8ca63b14fba8078980fy4ABpU4hSONMAyO  127.0.0.1   2021-09-20T15:32:27+02:00  ✔️       v1.2.0-openwrt-openwrt                                                 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Logs show me now an error :
time="20-09-2021 15:32:25" level=error msg="Failed to notify(sent: false): <nil>"

time="20-09-2021 15:32:14" level=warning msg="SIGTERM received, shutting down"
time="20-09-2021 15:32:14" level=info msg="Crowdsec engine shutting down"
time="20-09-2021 15:32:14" level=info msg="Killing parser routines"
time="20-09-2021 15:32:15" level=info msg="Bucket routine exiting"
time="20-09-2021 15:32:16" level=info msg="killing all plugins"
time="20-09-2021 15:32:16" level=info msg="serve: shutting down api server"
time="20-09-2021 15:32:16" level=info msg="push tomb is dying, sending cache (0 elements) before exiting"
time="20-09-2021 15:32:16" level=warning msg="Crowdsec service shutting down"
time="20-09-2021 15:32:16" level=info msg="Crowdsec v1.2.0-openwrt-openwrt"
time="20-09-2021 15:32:16" level=info msg="Loading prometheus collectors"
time="20-09-2021 15:32:16" level=info msg="Loading CAPI pusher"
time="20-09-2021 15:32:16" level=info msg="start crowdsec api push (interval: 30s)"
time="20-09-2021 15:32:16" level=info msg="start crowdsec api pull (interval: 2h)"
time="20-09-2021 15:32:16" level=info msg="start crowdsec api send metrics (interval: 30m)"
time="20-09-2021 15:32:16" level=info msg="last CAPI pull is newer than 1h30, skip."
time="20-09-2021 15:32:16" level=info msg="Loading grok library /etc/crowdsec/patterns"
time="20-09-2021 15:32:25" level=info msg="Loading enrich plugins"
time="20-09-2021 15:32:25" level=info msg="Successfully registered enricher 'GeoIpCity'"
time="20-09-2021 15:32:25" level=info msg="Successfully registered enricher 'GeoIpASN'"
time="20-09-2021 15:32:25" level=info msg="Successfully registered enricher 'IpToRange'"
time="20-09-2021 15:32:25" level=info msg="Successfully registered enricher 'reverse_dns'"
time="20-09-2021 15:32:25" level=info msg="Successfully registered enricher 'ParseDate'"
time="20-09-2021 15:32:25" level=info msg="Loading parsers 6 stages"
time="20-09-2021 15:32:25" level=info msg="Loaded 2 parser nodes" file=/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
time="20-09-2021 15:32:25" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/iptables-logs.yaml
time="20-09-2021 15:32:25" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
time="20-09-2021 15:32:25" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
time="20-09-2021 15:32:25" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
time="20-09-2021 15:32:25" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/parsers/s02-enrich/whitelists.yaml
time="20-09-2021 15:32:25" level=info msg="Loaded 7 nodes, 3 stages"
time="20-09-2021 15:32:25" level=info msg="Loading postoverflow Parsers"
time="20-09-2021 15:32:25" level=info msg="Loaded 0 nodes, 0 stages"
time="20-09-2021 15:32:25" level=info msg="Loading 2 scenario files"
time="20-09-2021 15:32:25" level=info msg="Adding leaky bucket" cfg=silent-fire file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf
time="20-09-2021 15:32:25" level=info msg="Adding leaky bucket" cfg=solitary-bush file=/etc/crowdsec/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf_user-enum
time="20-09-2021 15:32:25" level=info msg="Adding leaky bucket" cfg=twilight-night file=/etc/crowdsec/scenarios/iptables-scan-multi_ports.yaml name=crowdsecurity/iptables-scan-multi_ports
time="20-09-2021 15:32:25" level=warning msg="Loaded 3 scenarios"
time="20-09-2021 15:32:25" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="20-09-2021 15:32:25" level=warning msg="No matching files for pattern /var/log/nginx/*.log" type=file
time="20-09-2021 15:32:25" level=warning msg="No matching files for pattern ./tests/nginx/nginx.log" type=file
time="20-09-2021 15:32:25" level=warning msg="No matching files for pattern /var/log/auth.log" type=file
time="20-09-2021 15:32:25" level=warning msg="No matching files for pattern /var/log/syslog" type=file
time="20-09-2021 15:32:25" level=warning msg="No matching files for pattern /var/log/apache2/*.log" type=file
time="20-09-2021 15:32:25" level=error msg="Failed to notify(sent: false): <nil>"
time="20-09-2021 15:32:25" level=warning msg="Starting processing data"

I do not understand how to upgrade from 1.1.1 to 1.2.0 all 3 nodes of my testing platform !
I have two remote Debian and one central OpenWrt.
OpenWrt is the main routeur/firewall gateway.
Debian servers hosts Nginx and NextCloud.

The OpenWrt upgrade may have been problematic because of my postinstall script, where I have redone the main local api register…
I have a made a backup with cscli config backup on it before testing the package.

The Debian are just standard Debian packages which were upgraded via apt !
But looks like to been no more recognised any more as already validated on central CrowdSec !?

Where and what I do it wrong ?

Checked again the How to set up a CrowdSec multi-server installation - The open-source & multiplayer security solution

I clean the machines list just keeping the localhost (latest registered by faulty multiples cscli machines add -a)

redone the (with changing to my own OpenWrt IP !)

sudo cscli lapi register -u http://10.0.0.1:8080

Then OpenWrt seen again the servers :

root@LPM:~/custom/1.2.0# cscli machines list
-------------------------------------------------------------------------------------------------------------------------
 NAME                                              IP ADDRESS  LAST UPDATE                STATUS  VERSION                
-------------------------------------------------------------------------------------------------------------------------
 db3e872e345f48848d0d85ab5c529947GWkbyXJtyNnJziiS  127.0.0.1   2021-09-20T16:22:51+02:00  ✔️       v1.2.0-openwrt-openwrt 
 5244e15d7908402192135ac72b4acb10Xd83eOd8jss8u3Cs  10.4.2.16   2021-09-20T16:32:33+02:00  🚫                             
 2f827bc3cfb84cb0b1e59a5ae3f1492dLk6Uqo3ipKhWLWjU  10.4.2.159  2021-09-20T16:32:44+02:00  🚫                             
-------------------------------------------------------------------------------------------------------------------------

So I validate the two new comers !

root@LPM:~/custom/1.2.0# cscli machines validate 5244e15d7908402192135ac72b4acb10Xd83eOd8jss8u3Cs
INFO[20-09-2021 04:33:28 PM] machine '5244e15d7908402192135ac72b4acb10Xd83eOd8jss8u3Cs' validated successfuly 
root@LPM:~/custom/1.2.0# cscli machines validate 2f827bc3cfb84cb0b1e59a5ae3f1492dLk6Uqo3ipKhWLWjU
INFO[20-09-2021 04:33:34 PM] machine '2f827bc3cfb84cb0b1e59a5ae3f1492dLk6Uqo3ipKhWLWjU' validated successfuly

Restarted the servers crowdsec service…

root@LPM:~/custom/1.2.0# cscli machines list
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 NAME                                              IP ADDRESS  LAST UPDATE                STATUS  VERSION                                                                
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 db3e872e345f48848d0d85ab5c529947GWkbyXJtyNnJziiS  127.0.0.1   2021-09-20T16:22:51+02:00  ✔️       v1.2.0-openwrt-openwrt                                                 
 5244e15d7908402192135ac72b4acb10Xd83eOd8jss8u3Cs  10.4.2.16   2021-09-20T16:34:22+02:00  ✔️       v1.2.0-debian-pragmatic-linux-0ecfe7568790a15791011da27eb24e96e7d4a39f 
 2f827bc3cfb84cb0b1e59a5ae3f1492dLk6Uqo3ipKhWLWjU  10.4.2.159  2021-09-20T16:34:25+02:00  ✔️       v1.2.0-debian-pragmatic-linux-0ecfe7568790a15791011da27eb24e96e7d4a39f 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
root@LPM:~/custom/1.2.0# cscli metrics
INFO[20-09-2021 04:39:16 PM] Local Api Metrics:                           
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | POST   |    1 |
| /v1/decisions/stream | GET    |  100 |
| /v1/watchers         | POST   |    2 |
| /v1/watchers/login   | POST   |   24 |
+----------------------+--------+------+
INFO[20-09-2021 04:39:16 PM] Local Api Machines Metrics:                  
+--------------------------------------------------+------------+--------+------+
|                     MACHINE                      |   ROUTE    | METHOD | HITS |
+--------------------------------------------------+------------+--------+------+
| 5244e15d7908402192135ac72b4acb10Xd83eOd8jss8u3Cs | /v1/alerts | POST   |    1 |
+--------------------------------------------------+------------+--------+------+
INFO[20-09-2021 04:39:16 PM] Local Api Bouncers Metrics:                  
+------------------------------+----------------------+--------+------+
|           BOUNCER            |        ROUTE         | METHOD | HITS |
+------------------------------+----------------------+--------+------+
| cs-firewall-bouncer-yvYwOGfW | /v1/decisions/stream | GET    |  100 |
+------------------------------+----------------------+--------+------+

Where and what I have done it wrong ?

Is it okay to do this multiple time ?
Only needed at first install also ???

the console still not shows correct updated informations (version and status)…
It looks to communicate, the registered ID is unchanged, the online_api_credentials.yaml is the same and cscli capi status said sucess

banning looks to works :

root@LPM:~/custom/1.2.0# cscli alerts list
+-----+------------------------------+-----------------------------------+---------+--------------------------------+-----------+--------------------------------+
| ID  |            VALUE             |              REASON               | COUNTRY |               AS               | DECISIONS |           CREATED AT           |
+-----+------------------------------+-----------------------------------+---------+--------------------------------+-----------+--------------------------------+
| 715 | Ip:167.94.138.59             | crowdsecurity/http-bad-user-agent | US      |                                | ban:1     | 2021-09-20 16:37:22.36034268   |
|     |                              |                                   |         |                                |           | +0200 +0200                    |
| 714 | crowdsec/community-blocklist | update : +559/-0 IPs              |         |                                | ban:559   | 2021-09-20 16:11:38 +0200      |
|     |                              |                                   |         |                                |           | +0200                          |
root@LPM:~/custom/1.2.0# cscli metrics
INFO[20-09-2021 04:53:18 PM] Local Api Metrics:                           
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    |    1 |
| /v1/alerts           | POST   |    1 |
| /v1/decisions/stream | GET    |  184 |
| /v1/watchers         | POST   |    2 |
| /v1/watchers/login   | POST   |   25 |
+----------------------+--------+------+
INFO[20-09-2021 04:53:18 PM] Local Api Machines Metrics:                  
+--------------------------------------------------+------------+--------+------+
|                     MACHINE                      |   ROUTE    | METHOD | HITS |
+--------------------------------------------------+------------+--------+------+
| 5244e15d7908402192135ac72b4acb10Xd83eOd8jss8u3Cs | /v1/alerts | POST   |    1 |
| db3e872e345f48848d0d85ab5c529947GWkbyXJtyNnJziiS | /v1/alerts | GET    |    1 |
+--------------------------------------------------+------------+--------+------+
INFO[20-09-2021 04:53:18 PM] Local Api Bouncers Metrics:                  
+------------------------------+----------------------+--------+------+
|           BOUNCER            |        ROUTE         | METHOD | HITS |
+------------------------------+----------------------+--------+------+
| cs-firewall-bouncer-yvYwOGfW | /v1/decisions/stream | GET    |  184 |
+------------------------------+----------------------+--------+------+

servers are looks healthy :

admin@myNextCloud:~$ cscli metrics
INFO[20-09-2021 04:51:28 PM] Acquisition Metrics:                         
+--------------------------------------------------+------------+--------------+----------------+------------------------+
|                      SOURCE                      | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/apache2/nc-access.log              |        311 |          311 | -              | -                      |
| journalctl:journalctl-_SYSTEMD_UNIT=sshd.service |          1 | -            |              1 | -                      |
+--------------------------------------------------+------------+--------------+----------------+------------------------+
INFO[20-09-2021 04:51:28 PM] Parser Metrics:                              
+----------------------------------+------+--------+----------+
|             PARSERS              | HITS | PARSED | UNPARSED |
+----------------------------------+------+--------+----------+
| child-crowdsecurity/apache2-logs |  311 |    311 | -        |
| child-crowdsecurity/http-logs    |  933 |    623 |      310 |
| crowdsecurity/apache2-logs       |  311 |    311 | -        |
| crowdsecurity/dateparse-enrich   |  311 |    311 | -        |
| crowdsecurity/geoip-enrich       |  311 |    311 | -        |
| crowdsecurity/http-logs          |  311 |    309 |        2 |
| crowdsecurity/non-syslog         |  311 |    311 | -        |
| crowdsecurity/syslog-logs        |    1 |      1 | -        |
| crowdsecurity/whitelists         |  311 |    311 | -        |
+----------------------------------+------+--------+----------+
admin@myREVERSE:~$ cscli metrics
INFO[20-09-2021 04:51:35 PM] Buckets Metrics:                             
+--------------------------------------+---------------+-----------+--------------+--------+---------+
|                BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-bad-user-agent    | -             |         1 |            1 |      2 | -       |
| crowdsecurity/http-crawl-non_statics | -             | -         |            1 |      2 |       1 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[20-09-2021 04:51:35 PM] Acquisition Metrics:                         
+--------------------------------+------------+--------------+----------------+------------------------+
|             SOURCE             | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+--------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/nginx/access.log |        377 |          364 |             13 |                      4 |
+--------------------------------+------------+--------------+----------------+------------------------+
INFO[20-09-2021 04:51:35 PM] Parser Metrics:                              
+--------------------------------+------+--------+----------+
|            PARSERS             | HITS | PARSED | UNPARSED |
+--------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs  | 1092 |    730 |      362 |
| child-crowdsecurity/nginx-logs |  390 |    364 |       26 |
| crowdsecurity/dateparse-enrich |  364 |    364 | -        |
| crowdsecurity/geoip-enrich     |  364 |    364 | -        |
| crowdsecurity/http-logs        |  364 |    362 |        2 |
| crowdsecurity/nginx-logs       |  377 |    364 |       13 |
| crowdsecurity/non-syslog       |  377 |    377 | -        |
| crowdsecurity/whitelists       |  364 |    364 | -        |
+--------------------------------+------+--------+----------+
1 Like

Okay, must be some time to been reported correctly !
Now version are fine and the today alerts are also shown…

Sorry for noise…

May be I have done some steps incorrectly, advice welcome ! :wink:

cscli -c /etc/crowdsec/config.yaml machines add --force "$(cat /var/lib/dbus/machine-id)" -a -f /etc/crowdsec/local_api_credentials.yaml

Yes this should be done at installation and only if your crowdsec agent is installed on the same machine than the local API.

1 Like

Okay, so I need to study another installation method for the OpenWrt Package.
More generic usage need to be managed.

I cannot use wizard.sh, because of missing commands, so I need to package them as requirements.
I can get then a little inspiration from the Debian post/pres remove/install script, look the OpenWrt packaging have some equivalents features.

I can do a simplest installation for now, by simply removing initial setup and let the user customize is CrowdSec for his own use.

requirements for wizard.sh on OpenWrt

install is already available : coreutils-install
whiptail is also already available
envsubst is still missing… so I test a package and made a PR:
envsubst : envsubst: initial package v1.2.0 by erdoukki · Pull Request #16664 · openwrt/packages · GitHub
find : find may be tweaked or needed from coreutils

Stay tuned

what do you think about this :

grep -q "login:" /etc/crowdsec/local_api_credentials.yaml
[ $? -eq 0 ] || cscli -c /etc/crowdsec/config.yaml machines add --force "$(cat /var/lib/dbus/machine-id)" -a -f /etc/crowdsec/local_api_credentials.yaml && echo already registered...

In case of a first installation, the default /etc/crowdsec/local_api_credentials.yaml will not contain login: nor password: strings and values.
In case of upgrade, because these 2 values will be present, the grep will return 1 and the cscli machines add will not been executed !

I also have to add a test on /var/lib/dbus/machine-id because it is present only since OpenWrt 21.02.x and absent in 19.07.x !
May be something like this :

[ -f /var/lib/dbus/machine-id ] && cscli -c /etc/crowdsec/config.yaml machines add "$(cat /var/lib/dbus/machine-id)" -a -f /etc/crowdsec/local_api_credentials.yaml || cscli -c /etc/crowdsec/config.yaml machines add -a -f /etc/crowdsec/local_api_credentials.yaml

May I also remove the --force ?

tweaked to :

[ -s /etc/crowdsec/online_api_credentials.yaml ] || cscli -c /etc/crowdsec/config.yaml capi register -f /etc/crowdsec/online_api_credentials.yaml && echo already registered online...