AI crawlers

AI crawlers are the user agents AI companies use to gather web content — for model training (e.g. GPTBot, ClaudeBot, Google-Extended) and for live retrieval when a user's question triggers web search (e.g. ChatGPT-User, Perplexity-User). Your robots.txt decides which of them may read your site.

Blocking all AI crawlers is a real strategic decision, not a default: it can keep your content out of both training data and live answers, which for most brands means invisibility exactly where buyers are asking. A deliberate policy usually allows retrieval agents on marketing content while controlling training access case by case.