---
name: robots-txt
title: robots.txt
category: discoverability
---

# robots.txt

robots.txt is still the only machine-readable place a site can state its crawl policy, and it is the first thing an agent fetches. A missing or invalid file means every crawler is guessing.

## When to use this skill

Use this when a site fails the "robots.txt" check and you need to implement it.

## Goal

Publish /robots.txt with clear crawl rules

## Pass criteria

Returns HTTP 200 with plain text containing at least one User-agent directive.

## Implementation

This skill covers the diagnosis. The step-by-step implementation prompt is part
of the paid tier and is served per scan by the API — see https://agent-ready.t0ols.com/llms-full.txt.

## Specifications

- https://www.rfc-editor.org/rfc/rfc9309

## Verifying

Re-run the scan against the site and confirm the check reports a pass:

    curl -s https://agent-ready.t0ols.com/api/scan \
      -H 'content-type: application/json' \
      -d '{"url":"https://example.com","enabledChecks":["robotsTxt"]}'