So I figured it while writing this question, but I decided to go ahead and post it anyway so it might help other people.
Create a file called /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml.local with the following content:
deny_log_prefix: "crowdsec: "
deny_log: true
Then restart the firewall-bouncer service: systemctl restart crowdsec-firewall-bouncer
Dropped packages will now be logged by nftables in /var/log/messages.
May 6 13:07:39 blue-server kernel: crowdsec: IN=eth0 OUT= MAC=XXX SRC=XXX DST=XXX LEN=52 TOS=0x02 PREC=0x00 TTL=108 ID=29312 DF PROTO=TCP SPT=21079 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0