robots.txt

robots.txt is a plain text file at the root of a domain (example.com/robots.txt) that tells crawlers, including search engines and AI bots, which parts of a site they may crawl.

Each rule names a user agent, such as Googlebot or OAI-SearchBot, and the paths it may or may not crawl. The file can also point to the XML sitemap.

Example

User-agent: * and Allow: / let every crawler in. Adding User-agent: GPTBot with Disallow: / blocks only OpenAI’s training crawler, while ChatGPT search can still find the site through OAI-SearchBot. See AI crawlers for the main bot names.

What robots.txt does not do

  • It does not remove a page from search results. Use a noindex tag for that, and keep the page crawlable so the tag can be read.

  • It is not access control. Well-behaved bots follow it, others may not.

  • It is not the same as llms.txt, which summarizes a site for language models instead of setting rules.

Framer creates a robots.txt for every published site, and by default it allows all search and AI crawlers. On Pro and Enterprise plans, a custom file can replace it through Static Files. Trueform reviews robots.txt, noindex settings and redirects before every launch and migration.

FAQ

Where do I find my robots.txt?

Add /robots.txt to your domain, for example trueform.agency/robots.txt. Every site that has one serves it at that exact path.

Can I edit robots.txt in Framer?

Yes. Framer generates one automatically, and on Pro and Enterprise plans you can upload a custom robots.txt with Static Files. It then replaces the generated file.

Know the term. Put it to work.

Book a 30-minute call with our founder, Cédric. We'll show you where it fits on your site, with a clear next step on scope and timeline.

TF emblem logo in 3D

Know the term. Put it to work.

Book a 30-minute call with our founder, Cédric. We'll show you where it fits on your site, with a clear next step on scope and timeline.

TF emblem logo in 3D

Know the term. Put it to work.

Book a 30-minute call with our founder, Cédric. We'll show you where it fits on your site, with a clear next step on scope and timeline.

TF emblem logo in 3D