My search engine is flagged as malicious :-(

crowdsec
1

Hello.

I run an independent internet search engine, https://search.marginalia.nu, and I do my own crawling, but for some reason my IP is being flagged as malicious.

I always crawl from the same IP (81.170.128.21), I respect robots.txt, I honor HTTP 429 responses and Crawl-Delay directives, the crawler always uses the same distinct user-agent [search.marginalia.nu] that vividly marks it as coming from my search engine, and on top of all this the code for the entire search engine is open source, and I only crawl a site once every 6-8 weeks.

I don’t understand what more I can do to avoid being flagged as malicious?

2

Hello @marginalia_nu !

What makes you believe that your IP is being flagged as malicious?

Best regards,