robots.txt
robots.txt is a plain text file at the root of a domain (example.com/robots.txt) that tells crawlers, including search engines and AI bots, which parts of a site they may crawl.
Each rule names a user agent, such as Googlebot or OAI-SearchBot, and the paths it may or may not crawl. The file can also point to the XML sitemap.
Example
User-agent: * and Allow: / let every crawler in. Adding User-agent: GPTBot with Disallow: / blocks only OpenAI’s training crawler, while ChatGPT search can still find the site through OAI-SearchBot. See AI crawlers for the main bot names.
What robots.txt does not do
It does not remove a page from search results. Use a noindex tag for that, and keep the page crawlable so the tag can be read.
It is not access control. Well-behaved bots follow it, others may not.
It is not the same as llms.txt, which summarizes a site for language models instead of setting rules.
Framer creates a robots.txt for every published site, and by default it allows all search and AI crawlers. On Pro and Enterprise plans, a custom file can replace it through Static Files. Trueform reviews robots.txt, noindex settings and redirects before every launch and migration.
FAQ
Where do I find my robots.txt?
Add /robots.txt to your domain, for example trueform.agency/robots.txt. Every site that has one serves it at that exact path.
Can I edit robots.txt in Framer?
Yes. Framer generates one automatically, and on Pro and Enterprise plans you can upload a custom robots.txt with Static Files. It then replaces the generated file.
Related terms
AI crawlers
AI crawlers are the bots AI companies use to read websites, either to show pages in AI search answers, to fetch a page a user asked about, or to collect training data. Examples are OAI-SearchBot, GPTBot, Claude-SearchBot, ClaudeBot and PerplexityBot.
llms.txt
llms.txt is a proposed standard for a plain Markdown file at the root of a website (/llms.txt) that gives large language models a short, curated summary of the site and links to its most important pages.
Website migration
A website migration is moving a website to a new platform, domain, structure or design, for example from WordPress or Webflow to Framer, while keeping content, search rankings and integrations intact.
Related at Trueform:
