---
title: "AI bot rules | robots.txt AI bot user agent rules GPTBot ClaudeBot | Agent-Ready Check"
description: "Crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and Google-Extended all read robots.txt, and each has its own published behaviour. Pass criteria: robots.txt names known AI crawlers, or carries a wildcard group that covers them."
locale: en
canonical: https://agent-ready.t0ols.com/checks/ai-bot-rules/
---

# AI bot rules

> Category: Bot Access Control

Crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and Google-Extended all read robots.txt, and each has its own published behaviour.

## Why it matters

Naming them turns an implicit default into a decision you made on purpose, and lets you treat search crawlers differently from training crawlers.

## Pass criteria

robots.txt names known AI crawlers, or carries a wildcard group that covers them.

## How to implement it

Goal: Add User-agent rules for AI crawlers like GPTBot, ClaudeBot, and Google-Extended

robots.txt names known AI crawlers, or carries a wildcard group that covers them.

The step-by-step implementation prompt is part of the paid tier. The free layer gives you the diagnosis, the evidence and the specification links.

Specifications: https://www.rfc-editor.org/rfc/rfc9309, https://developers.cloudflare.com/ai-crawl-control/
Implementation notes: https://agent-ready.t0ols.com/.well-known/agent-skills/ai-rules/SKILL.md