Bot Access Control
AI bot rules
Crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and Google-Extended all read robots.txt, and each has its own published behaviour.
Why it matters
Naming them turns an implicit default into a decision you made on purpose, and lets you treat search crawlers differently from training crawlers.
Pass criteria
robots.txt names known AI crawlers, or carries a wildcard group that covers them.
What the scanner records
Every request the check makes is recorded (URL, method, response status, relevant headers and a body preview) and shown under the check in your results.
Every request the check makes is recorded (URL, method, response status, relevant headers and a body preview) and shown under the check in your results.
How to implement it
Add User-agent rules for AI crawlers like GPTBot, ClaudeBot, and Google-Extended
robots.txt names known AI crawlers, or carries a wildcard group that covers them.
The step-by-step implementation prompt is part of the Pro plan.
See plansScan your site
See whether this check passes on your domain, along with the other 21.