Does robots.txt block AI crawlers?

Well-behaved crawlers follow it; it is a request, not an access control.

Which user agents should I list?

The ones the check reports, each with its own Disallow rules, or a single User-agent: * section.

AI Crawler Access

A robots.txt file tells crawlers which parts of a site they may fetch, and AI companies run their own crawlers such as GPTBot, ClaudeBot, and CCBot. This check reads a site’s robots.txt and shows whether each known AI crawler is allowed or blocked.

When to use this: You want to confirm which AI crawlers can read your site after changing robots.txt. Command prefix robotsai:

About the AI Crawler Access

Analyze a site’s robots.txt rules for AI crawlers such as GPTBot, ClaudeBot and CCBot.

FAQ

Does robots.txt block AI crawlers?: Well-behaved crawlers follow it; it is a request, not an access control.
Which user agents should I list?: The ones the check reports, each with its own Disallow rules, or a single User-agent: * section.