Discoverability

robots.txt

robots.txt is still the only machine-readable place a site can state its crawl policy, and it is the first thing an agent fetches. A missing or invalid file means every crawler is guessing.

Why it matters

Everything downstream builds on it: the sitemap reference, the AI crawler rules and the Content Signal all live in this file.

Pass criteria

Returns HTTP 200 with plain text containing at least one User-agent directive.

What the scanner records
Every request the check makes is recorded (URL, method, response status, relevant headers and a body preview) and shown under the check in your results.

How to implement it

Publish /robots.txt with clear crawl rules

Returns HTTP 200 with plain text containing at least one User-agent directive.

The step-by-step implementation prompt is part of the Pro plan.

See plans

www.rfc-editor.org SKILL.md

Scan your site

See whether this check passes on your domain, along with the other 21.

Run a scan

Related checks

Sitemap

A valid XML sitemap is served at /sitemap.xml, or declared with a Sitemap: directive in robots.txt.

Link headers

The homepage response carries Link headers with at least one agent-useful relation.

DNS for AI Discovery

ServiceMode SVCB/HTTPS records or a TXT index exist under the _agents namespace, ideally DNSSEC-signed.