AI crawlers
AI crawlers are the bots AI companies use to read websites, either to show pages in AI search answers, to fetch a page a user asked about, or to collect training data. Examples are OAI-SearchBot, GPTBot, Claude-SearchBot, ClaudeBot and PerplexityBot.
Most AI companies now run separate bots for separate jobs, and each one can be allowed or blocked on its own in robots.txt.
The main ones
OAI-SearchBot (OpenAI): finds pages for ChatGPT search. Blocking it keeps your pages out of ChatGPT search answers.
GPTBot (OpenAI): crawls content that may be used to train OpenAI’s models.
ChatGPT-User (OpenAI): visits a page when a user’s request needs it. OpenAI notes robots.txt rules may not apply.
Claude-SearchBot, ClaudeBot and Claude-User (Anthropic): search indexing, training data and user-triggered fetches.
Google-Extended (Google): a robots.txt token that controls whether content is used for Gemini models. It does not affect Google Search.
Allow search, decide on training
A common setup is to allow the search bots, so assistants can cite you, and make a separate decision about training bots. Blocking everything also removes you from AI answers.
Framer pre-renders every page to HTML, so crawlers that don’t run JavaScript still get the full text, and it also serves a markdown version to AI tools that ask for one. By default Framer’s robots.txt allows all crawlers. Trueform checks these settings and the llms.txt file on every site we launch.
FAQ
Should I block GPTBot?
That is a business decision about training data. Blocking GPTBot does not remove you from ChatGPT search; that is controlled by OAI-SearchBot.
Can AI crawlers read a Framer site?
Yes. Framer serves pre-rendered HTML and a markdown version of each optimized page, and its default robots.txt allows search and AI crawlers.
Related terms
robots.txt
robots.txt is a plain text file at the root of a domain (example.com/robots.txt) that tells crawlers, including search engines and AI bots, which parts of a site they may crawl.
llms.txt
llms.txt is a proposed standard for a plain Markdown file at the root of a website (/llms.txt) that gives large language models a short, curated summary of the site and links to its most important pages.
Generative Engine Optimization (GEO)
Generative Engine Optimization (GEO) is the practice of making content more likely to be used and cited by generative AI systems, such as ChatGPT, Claude, Gemini or Perplexity, when they write an answer.
AI visibility
AI visibility is how often, and how accurately, AI assistants such as ChatGPT, Claude, Gemini, Perplexity and Copilot mention, describe and cite your brand when people ask questions about your category.
Related at Trueform:
