TurtleGuard

Docs / Rules and filters

Known bots

Search engines and services pass without a check; fakes of them do not.

2 minute readUpdated 2026-11-07

Search robots and the services that build link previews do not need a browser check: they reach the site on their own. Only the real ones, though — a bot that merely calls itself Googlebot gets checked like everyone else.

Which bots are known

PurposeBots
SearchGooglebot, Bingbot, YandexBot, DuckDuckBot, Baiduspider, Applebot, PetalBot
Link previewsTelegramBot, Twitterbot, facebookexternalhit, LinkedInBot, WhatsApp, Pinterestbot
SEO servicesAhrefsBot, SemrushBot

How a real bot is told apart

  1. The User-Agent carries the bot's name.
  2. The address belongs to the bot owner's network.
  3. The reverse DNS record of the address leads to the owner's domain (for example googlebot.com), and the forward record of that name returns the same address.

The check runs in the background, so the very first request of a new robot is not confirmed yet. The result is remembered for a day. Telegram is checked by the owner's network.

Whom to let through

Domain → Settings → Additional → “Bots that skip the challenge”. By default every bot on the list passes. Untick the ones you do not need — for example SEO services, if they load the site.

No check does not mean no limits

Verified bots skip only the browser check. The deny list, limits, the WAF and your block rules still apply to them.

Custom rules have a Verified bot field: use it, say, to open a path to search engines only or to close it to them.

Block by client fingerprint

A JA4 fingerprint describes how a program sets up the TLS connection. A botnet usually shares one even when its User-Agent changes. In Settings → Additional → “Client fingerprints to block” you can list fingerprints in full or by prefix (t13d1516h2_*), up to 500. Such clients get 403, and repeated connections are dropped in the node kernel. Fingerprints are visible in the request log.

VPN apps

If the site hands out VPN subscriptions, apps like Happ, sing-box, Clash, Hiddify and v2ray cannot pass a browser check. Turn them on in Protection → VPN subscription apps: they skip the check, but only on subscription paths (by default /sub/*, /api/sub/* and token links) and only for plain GET requests with a rate limit. A browser posing as such an app gets 403.