---
title: "robots.txt | robots.txt for AI crawlers | Agent-Ready Check"
description: "robots.txt is still the only machine-readable place a site can state its crawl policy, and it is the first thing an agent fetches. A missing or invalid file means every crawler is guessing. Pass criteria: Returns HTTP 200 with plain text containing at least one User-agent directive."
locale: en
canonical: https://agent-ready.t0ols.com/checks/robots-txt/
---

# robots.txt

> Category: Discoverability

robots.txt is still the only machine-readable place a site can state its crawl policy, and it is the first thing an agent fetches. A missing or invalid file means every crawler is guessing.

## Why it matters

Everything downstream builds on it: the sitemap reference, the AI crawler rules and the Content Signal all live in this file.

## Pass criteria

Returns HTTP 200 with plain text containing at least one User-agent directive.

## How to implement it

Goal: Publish /robots.txt with clear crawl rules

Returns HTTP 200 with plain text containing at least one User-agent directive.

The step-by-step implementation prompt is part of the paid tier. The free layer gives you the diagnosis, the evidence and the specification links.

Specifications: https://www.rfc-editor.org/rfc/rfc9309
Implementation notes: https://agent-ready.t0ols.com/.well-known/agent-skills/robots-txt/SKILL.md