AI crawlers

AI crawlers are the bots AI companies use to read websites, either to show pages in AI search answers, to fetch a page a user asked about, or to collect training data. Examples are OAI-SearchBot, GPTBot, Claude-SearchBot, ClaudeBot and PerplexityBot.

Most AI companies now run separate bots for separate jobs, and each one can be allowed or blocked on its own in robots.txt.

The main ones

  • OAI-SearchBot (OpenAI): finds pages for ChatGPT search. Blocking it keeps your pages out of ChatGPT search answers.

  • GPTBot (OpenAI): crawls content that may be used to train OpenAI’s models.

  • ChatGPT-User (OpenAI): visits a page when a user’s request needs it. OpenAI notes robots.txt rules may not apply.

  • Claude-SearchBot, ClaudeBot and Claude-User (Anthropic): search indexing, training data and user-triggered fetches.

  • Google-Extended (Google): a robots.txt token that controls whether content is used for Gemini models. It does not affect Google Search.

Allow search, decide on training

A common setup is to allow the search bots, so assistants can cite you, and make a separate decision about training bots. Blocking everything also removes you from AI answers.

Framer pre-renders every page to HTML, so crawlers that don’t run JavaScript still get the full text, and it also serves a markdown version to AI tools that ask for one. By default Framer’s robots.txt allows all crawlers. Trueform checks these settings and the llms.txt file on every site we launch.

FAQ

Should I block GPTBot?

That is a business decision about training data. Blocking GPTBot does not remove you from ChatGPT search; that is controlled by OAI-SearchBot.

Can AI crawlers read a Framer site?

Yes. Framer serves pre-rendered HTML and a markdown version of each optimized page, and its default robots.txt allows search and AI crawlers.

Know the term. Put it to work.

Book a 30-minute call with our founder, Cédric. We'll show you where it fits on your site, with a clear next step on scope and timeline.

TF emblem logo in 3D

Know the term. Put it to work.

Book a 30-minute call with our founder, Cédric. We'll show you where it fits on your site, with a clear next step on scope and timeline.

TF emblem logo in 3D

Know the term. Put it to work.

Book a 30-minute call with our founder, Cédric. We'll show you where it fits on your site, with a clear next step on scope and timeline.

TF emblem logo in 3D