---
name: ai-rules
title: AI bot rules
category: botAccessControl
---

# AI bot rules

Crawlers such as GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and Google-Extended all read robots.txt, and each has its own published behaviour.

## When to use this skill

Use this when a site fails the "AI bot rules" check and you need to implement it.

## Goal

Add User-agent rules for AI crawlers like GPTBot, ClaudeBot, and Google-Extended

## Pass criteria

robots.txt names known AI crawlers, or carries a wildcard group that covers them.

## Implementation

This skill covers the diagnosis. The step-by-step implementation prompt is part
of the paid tier and is served per scan by the API — see https://agent-ready.t0ols.com/llms-full.txt.

## Specifications

- https://www.rfc-editor.org/rfc/rfc9309
- https://developers.cloudflare.com/ai-crawl-control/

## Verifying

Re-run the scan against the site and confirm the check reports a pass:

    curl -s https://agent-ready.t0ols.com/api/scan \
      -H 'content-type: application/json' \
      -d '{"url":"https://example.com","enabledChecks":["robotsTxtAiRules"]}'