Multi server: TLS handshake error

diq · May 22, 2023, 2:42pm

I posted this on discord but got nothing back. Hoping to get different eyes on here

I have a multi server set up using the LetsEncrypt certs on the LAPI host. All looks to be working, so far so good.

However I’m getting a lot of errors which confuse me, surely this error is terminal?

TLS handshake error … connection reset by peer

It looks like the LAPI host is having issues communicating with itself and the other machine however it seems to return that the connection is fine

LAPI host (10.0.0.245)

$ journalctl -u crowdsec -f

May 12 09:23:02 ip-10-0-0-245 crowdsec[323054]: 2023/05/12 09:23:02 http: TLS handshake error from 10.0.0.178:65518: write tcp 10.0.0.245:9090->10.0.0.178:65518: write: connection reset by peer
May 12 09:23:05 ip-10-0-0-245 crowdsec[323054]: 2023/05/12 09:23:05 http: TLS handshake error from 10.0.0.245:58638: write tcp 10.0.0.245:9090->10.0.0.245:58638: write: connection reset by peer
May 12 09:23:07 ip-10-0-0-245 crowdsec[323054]: 2023/05/12 09:23:07 http: TLS handshake error from 10.0.0.245:34570: write tcp 10.0.0.245:9090->10.0.0.245:34570: write: connection reset by peer
May 12 09:23:08 ip-10-0-0-245 crowdsec[323054]: 2023/05/12 09:23:08 http: TLS handshake error from 10.0.0.178:29480: write tcp 10.0.0.245:9090->10.0.0.178:29480: write: connection reset by peer
May 12 09:23:09 ip-10-0-0-245 crowdsec[323054]: 2023/05/12 09:23:09 http: TLS handshake error from 10.0.0.245:34572: write tcp 10.0.0.245:9090->10.0.0.245:34572: write: connection reset by peer

$ sudo cscli machines list

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                                               IP Address   Last Update            Status   Version                                                                  Auth Type   Last Heartbeat 
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX   10.0.0.245   2023-05-12T09:13:33Z   ✔️        v1.4.6-debian-pragmatic-linux-5f71037b40c498045e1b59923504469e2b8d0140   password    20s            
 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX   10.0.0.178   2023-05-12T09:13:10Z   ✔️        v1.4.6-debian-pragmatic-linux-5f71037b40c498045e1b59923504469e2b8d0140   password    43s            
─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

$ sudo cscli lapi status
INFO[12-05-2023 09:15:28] You can successfully interact with Local API (LAPI)

$ sudo cscli bouncers list

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                           IP Address   Valid   Last API pull          Type                        Version                                                             Auth Type 
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 xxxxxxxxxxx-FirewallBouncer    10.0.0.245   ✔️       2023-05-12T09:18:15Z   crowdsec-firewall-bouncer   v0.0.26-debian-pragmatic-fa0e7d25b886aea927c1c3323ba854b5751fc3d1   api-key   
 xxxx-HAProxy-FirewallBouncer   10.0.0.178   ✔️       2023-05-12T09:18:23Z   crowdsec-firewall-bouncer   v0.0.26-debian-pragmatic-fa0e7d25b886aea927c1c3323ba854b5751fc3d1   api-key   
 xxxxxxxxxxx-AWS_WAF_Bouncer    10.0.0.245   ✔️       2023-05-12T09:18:17Z   crowdsec-aws-waf-bouncer    v0.1.5-debian-pragmatic-7f5e78e25ffc7009ff835782ffbda189608c7f2c    api-key   
 xxxx-HAProxy-HAProxyBouncer    10.0.0.178   ✔️       2023-05-12T09:18:21Z   crowdsec-haproxy-bouncer    v1.0.0                                                              api-key   
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

config.yaml

common:
  daemonize: true
  pid_dir: /var/run/
  log_media: file
  log_level: info
  log_dir: /var/log/
  log_max_size: 20
  compress_logs: true
  log_max_files: 10
  working_dir: .
config_paths:
  config_dir: /etc/crowdsec/
  data_dir: /var/lib/crowdsec/data/
  simulation_path: /etc/crowdsec/simulation.yaml
  hub_dir: /etc/crowdsec/hub/
  index_path: /etc/crowdsec/hub/.index.json
  notification_dir: /etc/crowdsec/notifications/
  plugin_dir: /usr/lib/crowdsec/plugins/
crowdsec_service:
  acquisition_path: /etc/crowdsec/acquis.yaml
  acquisition_dir: /etc/crowdsec/acquis.d
  parser_routines: 1
cscli:
  output: human
  color: auto
db_config:
  log_level: info
  type: mysql
  user: crowdsec
  password: XXXXXX
  db_name: crowdsec
  host: 127.0.0.1
  port: 3306
#  use_wal: true
#  log_level: info
#  type: sqlite
#  db_path: /var/lib/crowdsec/data/crowdsec.db
  #max_open_conns: 100
  #user: 
  #password:
  #db_name:
  #host:
  #port:
  flush:
    max_items: 5000
    max_age: 7d
plugin_config:
  user: nobody # plugin process would be ran on behalf of this user
  group: nogroup # plugin process would be ran on behalf of this group
api:
  client:
#    insecure_skip_verify: false
    insecure_skip_verify: true
    credentials_path: /etc/crowdsec/local_api_credentials.yaml
  server:
    log_level: info
#    listen_uri: 127.0.0.1:9090
    listen_uri: 10.0.0.245:9090
    profiles_path: /etc/crowdsec/profiles.yaml
    console_path: /etc/crowdsec/console.yaml
    online_client: # Central API credentials (to push signals and receive bad IPs)
      credentials_path: /etc/crowdsec/online_api_credentials.yaml
    trusted_ips: # IP ranges, or IPs which can have admin API access
      - 127.0.0.1
      - 10.0.0.245
      - ::1
    tls:
      cert_file: /etc/letsencrypt/live/XXXX/fullchain.pem
      key_file: /etc/letsencrypt/live/XXXX/privkey.pem
prometheus:
  enabled: true
  level: full
  listen_addr: 127.0.0.1
  listen_port: 6060

=====
Machine IP 10.0.0.178

$ sudo cscli lapi status
INFO[12-05-2023 09:15:28] You can successfully interact with Local API (LAPI)

Any help would be greatly appreciated

iiAmLoz · May 22, 2023, 3:35pm

Do you have any firewall rules setup on the host?

cscli lapi status just tries to login it wont test the underlying TLS. So the connection works, could you also try updating the latest version? we implemented a retry function, it wont fixed the issue but may improve it.

iiAmLoz · May 22, 2023, 3:38pm

Just to note im asking about the firewall because I think it killing the keep alive protocol.

diq · May 22, 2023, 3:47pm

$ dpkg -l|grep crowdsec
ii  crowdsec                           1.5.1                                   amd64        Crowdsec - An open-source, lightweight agent to detect and respond to bad behaviors. It also automatically benefits from our global community-wide IP reputation database
ii  crowdsec-aws-waf-bouncer           0.1.5                                   amd64        AWS WAF bouncer for Crowdsec
ii  crowdsec-firewall-bouncer-iptables 0.0.27                                  amd64        Firewall bouncer for Crowdsec (iptables+ipset)

The host is running a few docker containers

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists src

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists src
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (2 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.internal  tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             match-set crowdsec-blacklists src
RETURN     all  --  anywhere             anywhere

Is the retry function in one of the yaml files? I’m using yaml.local - do I need to add something to yaml.local ?

iiAmLoz · May 23, 2023, 10:16am

The retry function is out of the box for crowdsec itself. However, it seems your server or firewall is closing the connections hence the reset by peer. Is the box hosted on cloud infra? EG is there no other firewall infront?

Further more to stop the connection being classed as idle from CrowdSec sends a heartbeat crowdsec/heartbeat.go at 534328ca30eec7731e7d8e9140bb67f80102500c · crowdsecurity/crowdsec · GitHub

You sure these are not the bouncers?

EDIT: I just realised when looking at heartbeat codes these are 1 minute yours are much closer, so this would be coming from the bouncers. Did you configure them to https?

diq · May 23, 2023, 11:52am

Both servers are EC2. I have just made security group rules for both IPs on both EC2s, just to make sure

LAPI host:

On the guest if I stop the firewall bouncer the errors continue. If I stop the haproxy bouncer they stop. So the error 10.0.0.245:9090->10.0.0.178:29480: write: connection reset by peer is definitely a haproxy bouncer issue

If I stop both bouncers on the LAPI host the error 10.0.0.245:9090->10.0.0.245:58638: write: connection reset by peer continues

diq · May 23, 2023, 11:53am

Guest:

(Discourse only allows me to upload 1 image per post)

diq · May 23, 2023, 12:07pm

Host crowdsec-firewall-bouncer.yaml

mode: iptables
pid_dir: /var/run/
update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
#log_level: debug
log_compression: true
log_max_size: 100
log_max_backups: 3
log_max_age: 30
#api_url: http://127.0.0.1:9090/
api_url: https://10.0.0.245:9090/
api_key: XXXXX
#insecure_skip_verify: false
insecure_skip_verify: true
#disable_ipv6: false
disable_ipv6: true
deny_action: DROP
deny_log: false
supported_decisions_types:
  - ban
#to change log prefix
#deny_log_prefix: "crowdsec: "
#to change the blacklists name
blacklists_ipv4: crowdsec-blacklists
blacklists_ipv6: crowdsec6-blacklists
#type of ipset to use
ipset_type: nethash
#if present, insert rule in those chains
iptables_chains:
  - INPUT
  - FORWARD
  - DOCKER-USER

## nftables
nftables:
  ipv4:
    enabled: true
    set-only: false
    table: crowdsec
    chain: crowdsec-chain
  ipv6:
    enabled: true
    set-only: false
    table: crowdsec6
    chain: crowdsec6-chain
# packet filter
pf:
  # an empty string disables the anchor
  anchor_name: ""

prometheus:
  enabled: true
  listen_addr: 127.0.0.1
  listen_port: 60601

diq · May 23, 2023, 12:22pm

guest haproxy.cfg

global
  log /dev/log local0
  log /dev/log local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

  maxcompcpuusage 95

  # HAProxy CORS Lua library
  # https://www.haproxy.com/blog/enabling-cors-in-haproxy/
  lua-load /etc/haproxy/cors.lua

  tune.ssl.default-dh-param 2048

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

  # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHA>
  ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

  # Crowdsec bouncer >>>
  ## On some systems (we only identified the issue with a custom build on centos 6), haproxy cannot validate the certificate of the captcha service.
  ## If you see an unexplained 503 error in haproxy logs, uncomment this line.
  httpclient.ssl.verify none # Beware that this is disabling http client ssl verification and is only a temp workaround
  httpclient.resolvers.id captcha_dns_resolver # Tell the lua httpclient to use this DNS resolver. Replace with your own resolver if you already have one.
  lua-prepend-path /usr/lib/crowdsec/lua/haproxy/?.lua
  lua-load /usr/lib/crowdsec/lua/haproxy/crowdsec.lua # path to crowdsec.lua
  setenv CROWDSEC_CONFIG /etc/crowdsec/bouncers/crowdsec-haproxy-bouncer.conf # path to crowdsec bouncer configuration file
  # Crowdsec bouncer <<<
####
defaults
  log global
  mode http
  option httplog
  option dontlognull
  option forwardfor
  timeout connect 5s
  timeout client 50s
  timeout server 50s

  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

  # Maximum time to receive complete HTTP request headers - slowloris attacks
  timeout http-request 10s

####
frontend default
  bind *:80
  mode http
  log global
  option httplog
  option httpslog
  maxconn 2000
  option forwardfor

  # ACME challenge ACL
  acl is_acme_challenge path_beg /.well-known/acme-challenge/

  # Redirect non-ACME challenges to HTTPS
  http-request redirect scheme https code 301 if !{ ssl_fc } !is_acme_challenge
  http-request set-header X-Forwarded-Proto "https"
  http-request set-header X-Forwarded-Port "443"

  # ACME challenge backend
  use_backend certbot if is_acme_challenge

  # HSTS header
  http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains, preload"

  # ACME challenge backend
  use_backend certbot if is_acme_challenge

####
frontend ssl
  bind *:443 ssl crt /etc/ssl/haproxy/mydomain.com.pem
  log global
  maxconn  2000
  option forwardfor
  http-request set-header X-Forwarded-Proto https

  # Preventing IP spoofing
  http-request del-header X-Forwarded-For

  # Cloudfront IP forwarding
  http-request capture req.hdr(X-Forwarded-For) len 15
  http-request set-src hdr(True-Client-IP)

  # remove the Server header from responses
  http-response set-header Server webserver

  # minimise the amount of information you give out about your server
  http-response del-header X-Powered-By
  http-response del-header X-Application-Context

  # Permissions-Policy
  http-response set-header Permissions-Policy "interest-cohort=(), accelerometer=(), autoplay=(), battery=(), camera=(), gyroscope=(), microphone=()"

  # Add Expires header
  http-response set-header Expires %[date(7200),http_date]

  # Content-Security-Policy
  http-response set-header Content-Security-Policy "object-src 'none'; base-uri 'self'; style-src 'unsafe-inline' *.mydomain.com; frame-ancestors 'none'; script-src 'unsafe-inline' *.mydomain.com *.hcaptcha.com"

  # Domains
  acl xx hdr(host) -i xx.mydomain.com
  acl xxx hdr(host) -i xxx.mydomain.com
  acl xxxx hdr(host) -i xxxx.mydomain.com

  # Rewrite https://domain.com/x and /x/ to https://domain.com/ # app loop bug
  acl xx_x_path path_beg /x
  acl xxx_x_path hdr_beg(host) -i xxx.mydomain.com path_beg /x
  acl xxxx_x_path hdr_beg(host) -i xxxx.mydomain.com path_beg /x
  http-request redirect location /xxxx code 302 if { hdr(host) -i xx.mydomain.com } xx_x_path { path_reg ^/x/?$ }
  http-request redirect location /xxxx code 302 if { hdr(host) -i xxx.mydomain.com } xxx_x_path { path_reg ^/x/?$ }
  http-request redirect location /xxxx code 302 if { hdr(host) -i xxxx.mydomain.com } xxxx_x_path { path_reg ^/x/?$ }

  # Redirect
  acl root path -m str /
  http-request redirect code 302 location https://xx.mydomain.com/xxxxx if xx root
  http-request redirect code 302 location https://xxx.mydomain.com/xxxxx if xxx root
  http-request redirect code 302 location https://xxxx.mydomain.com/xxxxx if xxxx root

  # Crowdsec bouncer >>>
  stick-table type ip size 10k expire 30m # declare a stick table to cache captcha verifications
  http-request lua.crowdsec_allow # action to identify crowdsec remediation
  http-request track-sc0 src if { var(req.remediation) -m str "captcha-allow" } # cache captcha allow decision
  http-request redirect location %[var(req.redirect_uri)] if { var(req.remediation) -m str "captcha-allow" } # redirect to initial url
  http-request use-service lua.reply_captcha if { var(req.remediation) -m str "captcha" } # serve captcha template if remediation is captcha
  http-request use-service lua.reply_ban if { var(req.remediation) -m str "ban" } # serve ban template if remediation is ban
  # Crowdsec bouncer <<<

  # Backends
  use_backend backend if xx
  use_backend backend if xxx
  use_backend backend if xxxx

  default_backend backend-no-match
####    
backend backend
  mode http
  fullconn 2000
  log global
  option forwardfor
  http-response set-header Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
  http-response set-header Pragma "no-cache"
  http-request set-header Host xx.xx.mydomain.com

  # Cloudfront IP forwarding
  http-request set-header X-Real-IP %[src]

  # https://dannytsang.co.uk/securing-haproxy-headers/
  http-response add-header X-XSS-Protection "1; mode=block"
  http-response set-header X-Frame-Options "SAMEORIGIN"
  http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
  http-response add-header X-Content-Type-Options "nosniff"
  http-response set-header Referrer-Policy strict-origin

  # https://stackoverflow.com/questions/66263928/how-do-i-secure-cookies-in-haproxy-2-2-using-an-http-response-line
  http-response replace-header Set-Cookie ^((?:.(?!\ [Ss]ecure))*)$ \1;\ SameSite=None;\ Secure

  # https://www.haproxy.com/blog/how-to-secure-cookies-using-haproxy-enterprise/
  http-after-response replace-header Set-Cookie '(^((?!(?i)httponly).)*$)' "\1; HttpOnly"

  # prevent directory traversal
  http-request deny if { path_reg -i "/\.\./" }

  server backend xx.xx.mydomain.com:443 check ssl verify none

  # Crowdsec bouncer >>>
  # define a backend for the captcha provider to allow DNS resolution
backend captcha_verifier
   server hcaptcha_verifier hcaptcha.com:443 check

  #This is required to allow the lua code to perform DNS resolution
resolvers captcha_dns_resolver
  nameserver ns1 1.1.1.1:53 #You can change this to your own DNS resolver or another server

####
backend backend-no-match
  http-request deny deny_status 400

####
backend certbot
  mode http
  log global
  server local 127.0.0.1:10081

####
listen stats
    bind :8800 ssl crt /etc/ssl/haproxy/xx.mydomain.com.pem
    mode http
    option httplog
    log global
    stats enable
    stats refresh 5s
    maxconn 10
    stats uri /stats
    stats hide-version
    stats auth stats:XXXXXXX

Guest crowdsec-haproxy-bouncer.conf

ENABLED=true
API_KEY=XXXXX
# haproxy
# path to community_blocklist.map
MAP_PATH=/var/lib/crowdsec/lua/haproxy/community_blocklist.map
# bounce for all type of remediation that the bouncer can receive from the local API
BOUNCING_ON_TYPE=all
FALLBACK_REMEDIATION=ban
REQUEST_TIMEOUT=3000
UPDATE_FREQUENCY=10
# live or stream
MODE=stream
# exclude the bouncing on those location
EXCLUDE_LOCATION=
#those apply for "ban" action
# /!\ REDIRECT_LOCATION and RET_CODE can't be used together. REDIRECT_LOCATION take priority over RET_CODE
# path to ban template
BAN_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/ban.html
REDIRECT_LOCATION=
RET_CODE=
#those apply for "captcha" action
# Captcha Secret Key
SECRET_KEY=XXXXX
# captcha Site key
SITE_KEY=XXXXX
# path to captcha template
CAPTCHA_TEMPLATE_PATH=/var/lib/crowdsec/lua/haproxy/templates/captcha.html
CAPTCHA_EXPIRATION=3600

iiAmLoz · May 24, 2023, 8:57am

I remember that currently the lua is hard coded to http so you will need to manually change the lua code.

Live location

github.com

crowdsecurity/cs-haproxy-bouncer/blob/031e1e37fea48a97d9e102f660b47027510c5a9f/lib/crowdsec.lua#L82


      
              for remediation, expiration in string.gmatch(remediation_exp, "(%w+),(%d+)") do
                  if expiration ~= nil and tonumber(expiration) >= time_now then
                      if remediation == "null" then
                          return nil
                      end
                      return remediation
                  end
              end
          end
          
          
local link = "http://" .. core.backends["crowdsec"].servers["crowdsec"]:get_addr() .. "/v1/decisions?ip=" .. source_ip
          
          
core.Debug("Fetching decision for ip="..source_ip)
          local response = core.httpclient():get{
              url=link,
              headers={
                  ["X-Api-Key"]={runtime.conf["API_KEY"]},
                  ["Connection"]={"keep-alive"},
                  ["User-Agent"]={"crowdsec-haproxy-bouncer/v1.0.0"}
              },
              timeout=2*60*1000

Stream location

github.com

crowdsecurity/cs-haproxy-bouncer/blob/031e1e37fea48a97d9e102f660b47027510c5a9f/lib/crowdsec.lua#L205


      
              end
          
          
    txn:set_var("req.remediation", remediation)
          end
          
          
-- Called from task
          -- load decisions from LAPI
          local function refresh_decisions(is_startup)
              core.Debug("Stream Query with startup "..tostring(is_startup))
              -- TODO: get protocol from config
              local link = "http://" .. core.backends["crowdsec"].servers["crowdsec"]:get_addr() .. "/v1/decisions/stream?startup=" .. tostring(is_startup)
          
          
    core.Debug("Start fetching decisions: startup="..tostring(is_startup))
              local response = core.httpclient():get{
                  url=link,
                  headers={
                      ["X-Api-Key"]={runtime.conf["API_KEY"]},
                      ["Connection"]={"keep-alive"},
                      ["User-Agent"]={"crowdsec-haproxy-bouncer/v1.0.0"}
                  },
                  timeout=2*60*1000

Just to note I didnt see the backend for crowdsec within the config?

diq · May 24, 2023, 9:47am

Oops, looks like I failed to copy/paste the backend for crowdsec in. It’s there

backend crowdsec
  # define a backend for crowdsec to allow DNS resolution
  # replace 127.0.0.1:8080 by the listen URI of the crowdsec local API
  # server crowdsec 127.0.0.1:9999 check
  # server crowdsec 10.0.0.245:9090 check ssl verify none
  server crowdsec xx.xx.mydomain.com:9090 check ssl verify none
  # Crowdsec bouncer <<<

I’m not sure that hacking the lua code is a long term solution. Will this hold across updates?

diq · May 24, 2023, 10:01am

After editing crowdsec.lua and restarting haproxy I’m getting the exact same errors on the LAPI host.

Am I better off stripping all the TLS out of crowdsec and finding an encrypted way to forward the LAPI port? If so do you have any advice on the best way to do this?

iiAmLoz · May 24, 2023, 10:17am

Personally I never like the application to do TLS, I always offload it to haproxy or nginx but thats my own personal preference.

diq · May 24, 2023, 10:21am

Is there any documentation for doing this? Preferably in haproxy so there’s less to maintain

iiAmLoz · May 24, 2023, 10:28am

There not direct documentation, but I showcase here in multi server workshop

diq · May 24, 2023, 10:30am

Could I use ssh tunnelling?

iiAmLoz · May 24, 2023, 11:57am

Yes I never tried but it just the same as exposing any port in that manner.

diq · May 24, 2023, 12:02pm

Thanks for your help

diq · May 24, 2023, 1:39pm

Hang on - there’s still a problem…

On the LAPI host I’ve set up haproxy on 9091 to do ssl termination and pass through to crowdsec. The original errors in journalctl are no longer present

haproxy.cfg

frontend crowdsec
  bind *:9091 ssl crt /etc/ssl/haproxy/my.domain.pem
  mode http
  option httplog

  default_backend crowdsec

backend crowdsec
  mode http
  server local 10.0.0.245:9090 check

I can connect the guest machine and register the guest iptables bouncer but there’s still problems with the guest haproxy bouncer

Guest haproxy.cfg

backend crowdsec
   server crowdsec 10.0.0.245:9090 check ssl verify none

Guest haproxy.log

haproxy[65299]: [alert] 143/131645 (65299) : Got error fetching decisions from Crowdsec: 502 (<html><body><h1>502 Bad Gateway</h1>.The server returned an invalid or incomplete response..</body></html>.)
haproxy[65299]: -:- [24/May/2023:13:16:45.000] <HTTPCLIENT> <HTTPCLIENT>/<HTTPCLIENT> 2/0/4/-1/4 502 209 - - SH-- 0/0/0/0/3 0/0 {} "GET http://10.0.0.245:9091/v1/decisions/stream?startup=true HTTP/1.1"
Got error fetching decisions from Crowdsec: 502 (<html><body><h1>502 Bad Gateway</h1>.The server returned an invalid or incomplete response..</body></html>.)

I’ve undone the changes to crowdsec.lua

Is the only way to get haproxy working in a multi server setup to edit crowdsec.lua?? Or am I missing something? Surely hacking the crowdsec.lua isn’t going to survive updates?

diq · May 24, 2023, 1:44pm

Even if I edit crowdsec.lua I still get

haproxy[65804]: -:- [24/May/2023:13:42:01.065] <HTTPCLIENT> <HTTPCLIENT>/<HTTPSCLIENT> 2/0/-1/-1/3 503 217 - - SC-- 1/0/0/0/3 0/0 {} "GET https://10.0.0.245:9091/v1/decisions/stream?startup=true HTTP/1.1"
haproxy[65804]: Got error fetching decisions from Crowdsec: 503 (<html><body><h1>503 Service Unavailable</h1>.No server is available to handle this request..</body></html>.)
haproxy[65804]: [alert] 143/134201 (65804) : Got error fetching decisions from Crowdsec: 503 (<html><body><h1>503 Service Unavailable</h1>.No server is available to handle this request..</body></html>.)
haproxy[65804]: Got error fetching decisions from Crowdsec: 503 (<html><body><h1>503 Service Unavailable</h1>.No server is available to handle this request..</body></html>.)

Topic		Replies	Views
Signature is invalid	2	627	March 29, 2021
Too many open files while talking to LAPI crowdsec	5	852	September 7, 2021
New user, my feedbacks and problems crowdsec	10	1237	September 7, 2021
CrowdSec on my Ubuntu 20.04LTS Unifi Controller crowdsec	5	1673	December 17, 2021
127.0.0.1:8080: connect: connection refused crowdsec	1	1366	February 28, 2022

Multi server: TLS handshake error

Related Topics