New user, my feedbacks and problems

Foxinou35 · August 23, 2021, 5:36pm

Hi guys !
French guy here

I am installing Crowdsec for my company.

Got several problems

Sometimes at first startup, CS does not register with CAPI (error 500)

time=“23-08-2021 15:45:54” level=error msg=“capi pull top: get stream: Get "https://api.crowdsec.net/v2/decisions/stream?startup=true\”: received response status "500 Internal Server Error" when fetching https://api.crowdsec.net/v2/watchers/login"

After several restart, it succeeds… (effect : no line present with cscli alerts list)

Had problems with nginx versions on Debian with nginx-bouncer (problems with conflicts / dependancies with libnginx-mod-http-lua and nginx / nginx common.and “aio” module.
It ends up I moved to Ubuntu 20.04 and forgot aio
We have a not standard log format for nginx. We spent hours how to fix it, “playing” with glob patterns, etc etc (If someone can transform my log format to the correct nginx parser format, feel free to ask me
Cannot start the dashboard (trying on a fresh debian 11 on GCP)

image895×148 4.85 KB
By the way, crowdsec rules

PS : here is my nginx log format for those interested

log_format pjpo_timed_combined '$remote_addr - $remote_user [$time_local] ’
’ “$request” $status $body_bytes_sent ’
’ “ref=$http_referer” “ua=$http_user_agent” ’
’ “rLoc=$sent_http_location” ’
’ “reqt=$request_time” “respt=$upstream_response_time” “host=$host” “cache=$upstream_cache_status” “upstream=$upstream_addr” ’
’ “uheadt=$upstream_header_time” ’ ;

alteredCoder · August 24, 2021, 2:18pm

Hello @Foxinou35,

Thanks for your feedbacks!

After a quick investigation i could not found why you receive a 500 during the login (this might happen when the public IP of an existing crowdsec installation changed). If you want to have more precision about this bug, i invit you to chat with us on Gitter so we can debug your problem in private message.

About your nginx parser, if you can provide some exampe log lines we can help you to do the parser.

About the dashboard problem, what are the resouces (RAM/CPU) on the machine ?
What can also help us to debug this is by runing docker logs <docker_id> while the metabase docker is hanging.

Foxinou35 · August 24, 2021, 3:47pm

Hi
Here are the docker logs :

Machine is a test machine, e2 micro, 2vcpu, 1Gb memory, debian 11

Warning: environ value jdk-11.0.7+10 for key :java-version has been overwritten with 11.0.7
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
2021-08-24 15:33:24,877 INFO metabase.util :: Maximum memory available to JVM: 235.9 MB
2021-08-24 15:33:39,741 INFO util.encryption :: Saved credentials encryption is DISABLED for this Metabase instance. 🔓 
 For more information, see https://metabase.com/docs/latest/operations-guide/encrypting-database-details-at-rest.html
2021-08-24 15:33:48,545 INFO metabase.core :: 
Metabase v0.37.0.2 (ba7be09 release-x.37.x) 

Copyright © 2021 Metabase, Inc. 

Metabase Enterprise Edition extensions are NOT PRESENT.
2021-08-24 15:33:48,560 WARN metabase.core :: WARNING: You have enabled namespace tracing, which could log sensitive information like db passwords.
2021-08-24 15:33:48,561 INFO metabase.core :: Starting Metabase in STANDALONE mode
2021-08-24 15:33:48,625 INFO metabase.server :: Launching Embedded Jetty Webserver with config: 
 {:port 3000, :host "0.0.0.0"}

2021-08-24 15:33:48,711 INFO metabase.core :: Starting Metabase version v0.37.0.2 (ba7be09 release-x.37.x) ...
2021-08-24 15:33:48,762 INFO metabase.core :: System info:
 {"file.encoding" "UTF-8",
 "java.runtime.name" "OpenJDK Runtime Environment",
 "java.runtime.version" "11.0.7+10",
 "java.vendor" "AdoptOpenJDK",
 "java.vendor.url" "https://adoptopenjdk.net/",
 "java.version" "11.0.7",
 "java.vm.name" "OpenJDK 64-Bit Server VM",
 "java.vm.version" "11.0.7+10",
 "os.name" "Linux",
 "os.version" "5.10.0-8-cloud-amd64",
 "user.language" "en",
 "user.timezone" "GMT"}

2021-08-24 15:33:48,764 INFO metabase.plugins :: Loading plugins in /plugins...
2021-08-24 15:33:49,531 INFO util.files :: Extract file /modules/snowflake.metabase-driver.jar -> /plugins/snowflake.metabase-driver.jar
2021-08-24 15:33:53,019 INFO util.files :: Extract file /modules/googleanalytics.metabase-driver.jar -> /plugins/googleanalytics.metabase-driver.jar
2021-08-24 15:33:53,032 INFO util.files :: Extract file /modules/druid.metabase-driver.jar -> /plugins/druid.metabase-driver.jar
2021-08-24 15:33:53,261 INFO util.files :: Extract file /modules/redshift.metabase-driver.jar -> /plugins/redshift.metabase-driver.jar
2021-08-24 15:33:53,515 INFO util.files :: Extract file /modules/sparksql.metabase-driver.jar -> /plugins/sparksql.metabase-driver.jar
2021-08-24 15:33:58,516 INFO util.files :: Extract file /modules/sqlite.metabase-driver.jar -> /plugins/sqlite.metabase-driver.jar
2021-08-24 15:33:59,015 INFO util.files :: Extract file /modules/sqlserver.metabase-driver.jar -> /plugins/sqlserver.metabase-driver.jar
2021-08-24 15:33:59,036 INFO util.files :: Extract file /modules/oracle.metabase-driver.jar -> /plugins/oracle.metabase-driver.jar
2021-08-24 15:33:59,262 INFO util.files :: Extract file /modules/vertica.metabase-driver.jar -> /plugins/vertica.metabase-driver.jar
2021-08-24 15:33:59,267 INFO util.files :: Extract file /modules/bigquery.metabase-driver.jar -> /plugins/bigquery.metabase-driver.jar
2021-08-24 15:33:59,282 INFO util.files :: Extract file /modules/presto.metabase-driver.jar -> /plugins/presto.metabase-driver.jar
2021-08-24 15:33:59,508 INFO util.files :: Extract file /modules/mongo.metabase-driver.jar -> /plugins/mongo.metabase-driver.jar
2021-08-24 15:33:59,768 INFO util.files :: Extract file /modules/google.metabase-driver.jar -> /plugins/google.metabase-driver.jar
2021-08-24 15:34:02,031 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :redshift...e[0m
2021-08-24 15:34:05,016 INFO driver.impl :: e[34mRegistered abstract driver :sqle[0m  🚚
e[32mLoad driver :sql took 1.3 se[0m
2021-08-24 15:34:05,033 INFO driver.impl :: e[34mRegistered abstract driver :sql-jdbce[0m (parents: [:sql]) 🚚
e[32mLoad driver :sql-jdbc took 1.5 se[0m
2021-08-24 15:34:05,038 INFO driver.impl :: e[34mRegistered driver :postgrese[0m (parents: [:sql-jdbc]) 🚚
e[32mLoad driver :postgres took 3.2 se[0m
2021-08-24 15:34:05,278 INFO driver.impl :: e[34mRegistered driver :redshifte[0m (parents: [:postgres]) 🚚
2021-08-24 15:34:05,511 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :druid...e[0m
2021-08-24 15:34:05,512 INFO driver.impl :: e[34mRegistered driver :druide[0m  🚚
2021-08-24 15:34:05,773 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :hive-like...e[0m
2021-08-24 15:34:05,775 INFO driver.impl :: e[34mRegistered abstract driver :hive-likee[0m (parents: [:sql-jdbc]) 🚚
2021-08-24 15:34:05,776 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sparksql...e[0m
2021-08-24 15:34:05,777 INFO driver.impl :: e[34mRegistered driver :sparksqle[0m (parents: [:hive-like]) 🚚
2021-08-24 15:34:06,024 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :snowflake...e[0m
2021-08-24 15:34:06,027 INFO driver.impl :: e[34mRegistered driver :snowflakee[0m (parents: [:sql-jdbc]) 🚚
2021-08-24 15:34:06,032 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :google...e[0m
2021-08-24 15:34:06,034 INFO driver.impl :: e[34mRegistered abstract driver :googlee[0m  🚚
2021-08-24 15:34:06,261 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Oracle Driver due to required dependencies.e[0m Metabase requires the Oracle JDBC driver in order to connect to Oracle databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/oracle.html for more details.

2021-08-24 15:34:06,267 INFO plugins.dependencies :: Metabase Oracle Driver dependency {:class oracle.jdbc.OracleDriver} satisfied? false
2021-08-24 15:34:06,269 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Oracle Driver"]e[0m
2021-08-24 15:34:06,286 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :mongo...e[0m
2021-08-24 15:34:06,508 INFO driver.impl :: e[34mRegistered driver :mongoe[0m  🚚
2021-08-24 15:34:06,519 INFO plugins.dependencies :: Plugin 'Metabase BigQuery Driver' depends on plugin 'Metabase Google Drivers Shared Dependencies'
2021-08-24 15:34:06,521 INFO plugins.dependencies :: Metabase BigQuery Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? true
2021-08-24 15:34:06,525 WARN plugins.lazy-loaded-driver :: e[31mWarning: plugin manifest for :bigquery does not include connection propertiese[0m
2021-08-24 15:34:06,526 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :bigquery...e[0m
2021-08-24 15:34:06,528 INFO driver.impl :: e[34mRegistered driver :bigquerye[0m (parents: [:sql :google]) 🚚
2021-08-24 15:34:06,766 INFO plugins.dependencies :: Plugin 'Metabase Google Analytics Driver' depends on plugin 'Metabase Google Drivers Shared Dependencies'
2021-08-24 15:34:06,768 INFO plugins.dependencies :: Metabase Google Analytics Driver dependency {:plugin Metabase Google Drivers Shared Dependencies} satisfied? true
2021-08-24 15:34:06,770 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :googleanalytics...e[0m
2021-08-24 15:34:06,772 INFO driver.impl :: e[34mRegistered driver :googleanalyticse[0m (parents: [:google]) 🚚
2021-08-24 15:34:06,788 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlite...e[0m
2021-08-24 15:34:07,012 INFO driver.impl :: e[34mRegistered driver :sqlitee[0m (parents: [:sql-jdbc]) 🚚
2021-08-24 15:34:07,024 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :presto...e[0m
2021-08-24 15:34:07,031 INFO driver.impl :: e[34mRegistered driver :prestoe[0m (parents: [:sql]) 🚚
2021-08-24 15:34:07,261 DEBUG plugins.lazy-loaded-driver :: e[35mRegistering lazy loading driver :sqlserver...e[0m
2021-08-24 15:34:07,263 INFO driver.impl :: e[34mRegistered driver :sqlservere[0m (parents: [:sql-jdbc]) 🚚
2021-08-24 15:34:07,268 INFO plugins.dependencies :: e[31mMetabase cannot initialize plugin Metabase Vertica Driver due to required dependencies.e[0m Metabase requires the Vertica JDBC driver in order to connect to Vertica databases, but we can't ship it as part of Metabase due to licensing restrictions. See https://metabase.com/docs/latest/administration-guide/databases/vertica.html for more details.

2021-08-24 15:34:07,269 INFO plugins.dependencies :: Metabase Vertica Driver dependency {:class com.vertica.jdbc.Driver} satisfied? false
2021-08-24 15:34:07,270 INFO plugins.dependencies :: e[33mPlugins with unsatisfied deps: ["Metabase Vertica Driver" "Metabase Oracle Driver"]e[0m
2021-08-24 15:34:07,277 INFO driver.impl :: e[34mRegistered driver :h2e[0m (parents: [:sql-jdbc]) 🚚
2021-08-24 15:34:07,515 INFO driver.impl :: e[34mRegistered driver :mysqle[0m (parents: [:sql-jdbc]) 🚚
2021-08-24 15:34:07,533 INFO metabase.core :: Setting up and migrating Metabase DB. Please sit tight, this may take a minute...
2021-08-24 15:34:07,536 WARN metabase.db :: e[31mWARNING: Using Metabase with an H2 application database is not recommended for production deployments. For production deployments, we highly recommend using Postgres, MySQL, or MariaDB instead. If you decide to continue to use H2, please be sure to back up the database file regularly. For more information, see https://metabase.com/docs/latest/operations-guide/migrating-from-h2.htmle[0m
2021-08-24 15:34:07,766 INFO metabase.db :: e[36mVerifying h2 Database Connection ...e[0m
2021-08-24 15:34:07,772 INFO driver.impl :: e[33mInitializing driver :sql...e[0m
2021-08-24 15:34:07,778 INFO driver.impl :: e[33mInitializing driver :sql-jdbc...e[0m
2021-08-24 15:34:07,779 INFO driver.impl :: e[33mInitializing driver :h2...e[0m
2021-08-24 15:34:12,778 ERROR driver.util :: Database connection error
java.util.concurrent.TimeoutException: Timed out after 5.0 s
	at metabase.util$deref_with_timeout.invokeStatic(util.clj:315) ~[metabase.jar:?]
	at metabase.util$deref_with_timeout.invoke(util.clj:307) ~[metabase.jar:?]
	at metabase.util$do_with_timeout.invokeStatic(util.clj:321) ~[metabase.jar:?]
	at metabase.util$do_with_timeout.invoke(util.clj:318) ~[metabase.jar:?]
	at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:34) [metabase.jar:?]
	at metabase.driver.util$can_connect_with_details_QMARK_.doInvoke(util.clj:23) [metabase.jar:?]
	at clojure.lang.RestFn.invoke(RestFn.java:442) [metabase.jar:?]
	at clojure.lang.Var.invoke(Var.java:393) [metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798$fn__21801$fn__21802.invoke(db.clj:316) [metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798$fn__21801.invoke(db.clj:315) [metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798.invoke(db.clj:307) [metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798$fn__21799.invoke(db.clj:310) [metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798.invoke(db.clj:307) [metabase.jar:?]
	at metabase.db$setup_db_BANG__STAR_$fn__21837.invoke(db.clj:352) [metabase.jar:?]
	at metabase.util$do_with_us_locale.invokeStatic(util.clj:670) [metabase.jar:?]
	at metabase.util$do_with_us_locale.invoke(util.clj:656) [metabase.jar:?]
	at metabase.db$setup_db_BANG__STAR_.invokeStatic(db.clj:351) [metabase.jar:?]
	at metabase.db$setup_db_BANG__STAR_.invoke(db.clj:346) [metabase.jar:?]
	at metabase.db$setup_db_from_env_BANG_.invokeStatic(db.clj:365) [metabase.jar:?]
	at metabase.db$setup_db_from_env_BANG_.invoke(db.clj:358) [metabase.jar:?]
	at metabase.db$setup_db_BANG_$fn__21842.invoke(db.clj:375) [metabase.jar:?]
	at metabase.db$setup_db_BANG_.invokeStatic(db.clj:373) [metabase.jar:?]
	at metabase.db$setup_db_BANG_.invoke(db.clj:368) [metabase.jar:?]
	at metabase.core$init_BANG_.invokeStatic(core.clj:96) [metabase.jar:?]
	at metabase.core$init_BANG_.invoke(core.clj:75) [metabase.jar:?]
	at metabase.core$start_normally.invokeStatic(core.clj:140) [metabase.jar:?]
	at metabase.core$start_normally.invoke(core.clj:134) [metabase.jar:?]
	at metabase.core$_main.invokeStatic(core.clj:173) [metabase.jar:?]
	at metabase.core$_main.doInvoke(core.clj:167) [metabase.jar:?]
	at clojure.lang.RestFn.invoke(RestFn.java:397) [metabase.jar:?]
	at clojure.lang.AFn.applyToHelper(AFn.java:152) [metabase.jar:?]
	at clojure.lang.RestFn.applyTo(RestFn.java:132) [metabase.jar:?]
	at metabase.core.main(Unknown Source) [metabase.jar:?]
2021-08-24 15:34:13,037 ERROR metabase.core :: Metabase Initialization FAILED
java.lang.Exception: Timed out after 5.0 s
	at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:40) ~[metabase.jar:?]
	at metabase.driver.util$can_connect_with_details_QMARK_.doInvoke(util.clj:23) ~[metabase.jar:?]
	at clojure.lang.RestFn.invoke(RestFn.java:442) ~[metabase.jar:?]
	at clojure.lang.Var.invoke(Var.java:393) ~[metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798$fn__21801$fn__21802.invoke(db.clj:316) ~[metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798$fn__21801.invoke(db.clj:315) ~[metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798.invoke(db.clj:307) ~[metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798$fn__21799.invoke(db.clj:310) ~[metabase.jar:?]
	at metabase.db$fn__21789$verify_db_connection__21798.invoke(db.clj:307) ~[metabase.jar:?]
	at metabase.db$setup_db_BANG__STAR_$fn__21837.invoke(db.clj:352) ~[metabase.jar:?]
	at metabase.util$do_with_us_locale.invokeStatic(util.clj:670) ~[metabase.jar:?]
	at metabase.util$do_with_us_locale.invoke(util.clj:656) ~[metabase.jar:?]
	at metabase.db$setup_db_BANG__STAR_.invokeStatic(db.clj:351) ~[metabase.jar:?]
	at metabase.db$setup_db_BANG__STAR_.invoke(db.clj:346) ~[metabase.jar:?]
	at metabase.db$setup_db_from_env_BANG_.invokeStatic(db.clj:365) ~[metabase.jar:?]
	at metabase.db$setup_db_from_env_BANG_.invoke(db.clj:358) ~[metabase.jar:?]
	at metabase.db$setup_db_BANG_$fn__21842.invoke(db.clj:375) ~[metabase.jar:?]
	at metabase.db$setup_db_BANG_.invokeStatic(db.clj:373) ~[metabase.jar:?]
	at metabase.db$setup_db_BANG_.invoke(db.clj:368) ~[metabase.jar:?]
	at metabase.core$init_BANG_.invokeStatic(core.clj:96) ~[metabase.jar:?]
	at metabase.core$init_BANG_.invoke(core.clj:75) ~[metabase.jar:?]
	at metabase.core$start_normally.invokeStatic(core.clj:140) [metabase.jar:?]
	at metabase.core$start_normally.invoke(core.clj:134) [metabase.jar:?]
	at metabase.core$_main.invokeStatic(core.clj:173) [metabase.jar:?]
	at metabase.core$_main.doInvoke(core.clj:167) [metabase.jar:?]
	at clojure.lang.RestFn.invoke(RestFn.java:397) [metabase.jar:?]
	at clojure.lang.AFn.applyToHelper(AFn.java:152) [metabase.jar:?]
	at clojure.lang.RestFn.applyTo(RestFn.java:132) [metabase.jar:?]
	at metabase.core.main(Unknown Source) [metabase.jar:?]
Caused by: java.util.concurrent.TimeoutException: Timed out after 5.0 s
	at metabase.util$deref_with_timeout.invokeStatic(util.clj:315) ~[metabase.jar:?]
	at metabase.util$deref_with_timeout.invoke(util.clj:307) ~[metabase.jar:?]
	at metabase.util$do_with_timeout.invokeStatic(util.clj:321) ~[metabase.jar:?]
	at metabase.util$do_with_timeout.invoke(util.clj:318) ~[metabase.jar:?]
	at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:34) ~[metabase.jar:?]
	... 28 more
2021-08-24 15:34:13,277 INFO metabase.core :: Metabase Shutting Down ...
2021-08-24 15:34:13,278 INFO metabase.server :: Shutting Down Embedded Jetty Webserver
2021-08-24 15:34:13,767 INFO metabase.core :: Metabase Shutdown COMPLETE

Regarding nginx logs (anonymized), 2 sample lines

91.91.91.91 - - [24/Aug/2021:15:32:22 +0000]   "GET /wp-content/plugins/revslider/public/assets/fonts/revicons/revicons.woff?5510888 HTTP/1.1" 200 7536  "ref=https://www.toto.fr/wp-content/plugins/revslider/public/assets/css/rs6.css?ver=6.1.1" "ua=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"  "rLoc=-"  "reqt=0.004" "respt=0.004" "host=www.toto.fr" "cache=-" "upstream=10.216.124.44:9010"  "uheadt=0.004" 
93.93.93.93 - - [24/Aug/2021:15:32:22 +0000]   "GET /wp-content/uploads/sites/8356/2020/04/pellets-1.jpg HTTP/1.1" 200 148831  "ref=https://www.toto.fr/titi/nos-produits/" "ua=Mozilla/5.0 (Linux; Android 11; SM-A705FN) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Mobile Safari/537.36"  "rLoc=-"  "reqt=0.004" "respt=0.000" "host=www.tutu.fr" "cache=-" "upstream=10.216.124.44:9010"  "uheadt=0.000"

And last problem, regarding prometheus/grafana, the metric
cs_reader_hits_total

is not exposed by the prometheus exporter

root@toto:/var/log/nginx# curl -s http://127.0.0.1:9169/metrics|grep reader
root@toto:/var/log/nginx# curl -s http://127.0.0.1:9169/metrics|grep cs_
# HELP cs_bucket_created_total Total buckets were instanciated.
# TYPE cs_bucket_created_total counter
cs_bucket_created_total{name="crowdsecurity/http-bf-wordpress_bf"} 2544

Prom config :

prometheus:
  enabled: true
  level: full
  listen_addr: 0.0.0.0
  listen_port: 9169

Thanks a lot

alteredCoder · August 24, 2021, 4:03pm

Hello,

For the docker metabase, i think your machine is not powerful enough. The docker metabase image need between 1-2Gb of RAM to build (it’s mentionned here in the documentation).

For the nginx parser, i will try to propose you a GROK pattern that will match your log format.

About prometheus, nice catch! We just revamped the acquisition part and forgot this part We will fix this soon. Sorry for the inconvenience.

alteredCoder · August 26, 2021, 10:07am

Hello @Foxinou35,

Here is the grok pattern that you can use in your nginx parser:

%{IPORHOST:remote_addr} - (%{NGUSER:remote_user})? \[%{HTTPDATE:time_local}\]   "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent}  "ref=%{NOTDQUOTE:http_referer}" "ua=%{NOTDQUOTE:http_user_agent}"  "rLoc=%{NOTDQUOTE:http_location}"  "reqt=%{NOTDQUOTE:request_time}" "respt=%{NOTDQUOTE:response_time}" "host=%{NOTDQUOTE:host}" "cache=%{NOTDQUOTE:cache_status}" "upstream=%{IPORHOST:source_addr}:%{NUMBER:source_port}"  "uheadt=%{NOTDQUOTE:upstream_header_time}"

Foxinou35 · August 26, 2021, 10:34am

Thanks a lot @alteredCoder will try it !

Foxinou35 · August 26, 2021, 12:59pm

It seems to work, thanks a lot !

But when I copy/paste your GROK pattern to grokdebugguer.com for example, it is not working (this is a kind of service we used to try to match the pattern)

Is there a special grok “engine” in crowdsec ?

alteredCoder · August 26, 2021, 1:29pm

The grok engine is not special in crowdsec, but we might have some custom GROK pattern that are not defined in grokdebugger. For example, i’m not sure the NOTDQUOTE grok is predefined in grokdebugger. But you can add custom pattern in grok pattern, for NOTDQUOTE the correspondig regular expression is: [^"]*.

If you want to search other grok pattern that crowdsec use, they are defined here

Foxinou35 · August 31, 2021, 12:47pm

Hi !
Just to give updates about my 500 errors at startup.
I use Ansible to deploy my 4 instances of crowdsec. It seems that the 1st instance registered ok with CAPI, but not the others.
They have different logins/passwords generated by the installer I guess.

But I have clearly noticed on one of the server, that the cscli capi status said, the same IP was used or something like that.
I just restarted the service, and status said OK.

Obviously, our 4 nginx servers are “hidden” behind the same public IP.

Other question about the community blocklist. Every hour new IP are obtained from the community. Are they like “decisions” IP (lasting 4 hours) ? so we have always 64*4 (hours) ? is there a max number ?
Because I guess on your side, you should have thousand of active malicious IPs.
Can you tell us more about this ?

thibault · September 2, 2021, 3:41pm

Hello @Foxinou35 !

You should get this error only if the various machines share the same credentials to the central API. (see /etc/crowdsec/online_api_credentials.yaml). Can you let me know more about the setup ? We might want to address this issue

Yes, there is more to come about this (hopefully) next week. Currently (because false-positives and poisoning are the main threat for a project like us) the community blocklist that is distributed is short and relies on our honeypot reports. The new community blocklist includes a lot more IPs to be redistributed (based truly on the community reports) and is undergoing final testing before general availability !

Hope this helps, let us know

Foxinou35 · September 7, 2021, 1:47pm

Regarding the same credentials. I can confirm they don’t have the same credentials.
But they are deployed via ansible, almost at the same time, with same external IP.
So maybe your system is detecting that and it causes problem…

The Ansible use the install scripts (crowdsec + bouncer) and set up custom conf for nginx parser.
That’s all

Topic		Replies	Views
CrowdSec on my Ubuntu 20.04LTS Unifi Controller crowdsec	5	1670	December 17, 2021
\|\| Installation Not Successfull \|\| crowdsec	1	576	May 26, 2021
Crowdsec Stopped Working on my Raspberry Pi (ARM?) crowdsec	5	690	February 6, 2023
127.0.0.1:8080: connect: connection refused crowdsec	1	1361	February 28, 2022
Crowdsec failing to start after executing cscli explain crowdsec	2	1326	May 25, 2022

New user, my feedbacks and problems

Related Topics