My search engine is flagged as malicious :-(


I run an independent internet search engine,, and I do my own crawling, but for some reason my IP is being flagged as malicious.

I always crawl from the same IP (, I respect robots.txt, I honor HTTP 429 responses and Crawl-Delay directives, the crawler always uses the same distinct user-agent [] that vividly marks it as coming from my search engine, and on top of all this the code for the entire search engine is open source, and I only crawl a site once every 6-8 weeks.

I don’t understand what more I can do to avoid being flagged as malicious?

Hello @marginalia_nu !

What makes you believe that your IP is being flagged as malicious?

Best regards,