robots.txt

Robots.txt is a plain text file at the root of a website (for example example.com/robots.txt) that tells crawlers which URLs they may fetch. For AI search, this file decides whether the bots behind ChatGPT, Perplexity, Claude and Google can read your site and cite it.

What is robots.txt and how does it work?

Each host has one robots.txt file at its root, organized in groups: a User-agent line names the bot, followed by Disallow (do not fetch) and Allow (may fetch) lines. A crawler follows only the group whose name matches it most specifically (Google Search Central, accessed November 2026). If there is a Googlebot group, Googlebot ignores the * group.

To keep a page out of Google, use a noindex rule instead. Reputable crawlers obey robots.txt; others may not (Google, Introduction to robots.txt, accessed November 2026).

Robots.txt for AI bots: search bots vs training bots

A search bot reads pages so an AI product can link to them in answers. A training bot collects content to train models. According to each vendor’s documentation, blocking the training bot does not remove you from that vendor’s AI search results.

• OpenAI: sites that block OAI-SearchBot will not be shown in ChatGPT search answers; GPTBot is a separate, independent setting. Changes take about 24 hours to apply (OpenAI Crawlers, accessed November 2026).
• Anthropic: blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude’s search results (Anthropic Help Center, accessed November 2026).
• Perplexity: PerplexityBot is not used to crawl content for AI foundation models; Perplexity-User generally ignores robots.txt because a person requested the fetch (Perplexity Crawlers, accessed November 2026).
• Google: Google-Extended does not affect inclusion or ranking in Google Search; it only covers Gemini and Vertex AI (Google’s common crawlers, accessed November 2026). AI Overviews rely on Googlebot (Google, AI features, accessed November 2026).

For crawl volumes by vendor, see the analysis of 68.9 million AI crawler visits.

How the AI Search Checker scores robots.txt

The “AI bot access” group is worth 20 of 100 points and checks 14 bots. Search bots carry 14.5 points (Googlebot 4, Bingbot 3, OAI-SearchBot 3, PerplexityBot 2, Claude-SearchBot 1.5, meta-webindexer 1). User-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) carry 2 points, training bots (GPTBot, Google-Extended, ClaudeBot, Applebot-Extended) 3 points and facebookexternalhit 0.5 points.

Hypothetical example: a site that blocks the 4 training bots and allows the rest scores 20 minus 3, or 17 points. That is a valid choice.

How to fix robots.txt for AI bots

1. Open yourdomain.com/robots.txt and save a backup. Look for Disallow: / under a search bot or under User-agent: *.
2. Decide whether to allow or block training bots.
3. Edit the file using the template below and save it on your platform:
   • Yoast SEO: Yoast SEO, Tools, File editor (Yoast, accessed November 2026).
   • Rank Math: turn on Advanced Mode, then Rank Math SEO, General Settings, Edit robots.txt; delete any physical robots.txt in the root folder first (Rank Math, accessed November 2026).
   • Shopify: Online Store, Themes, Edit code, Add a new template, choose robots (Shopify Help Center, accessed November 2026).
   • Haravan, Sapo (Vietnamese ecommerce platforms): ask the platform’s support team.
   • Custom site: put the file at the host root and make sure it returns HTTP 200.
4. Check your work: open the robots.txt report in Google Search Console, click Request a recrawl (Google, accessed November 2026), then rerun the AI Search Checker.

# Search bots: allow
User-agent: Googlebot
User-agent: Bingbot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: meta-webindexer
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# Training bots: block (optional)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap.xml

The search bot group repeats the /wp-admin/ rules because a bot with its own group does not read the * group. On Shopify, add only the training bot group, after the default Liquid block. CCBot collects the open web archive of Common Crawl (accessed November 2026). The Sitemap line points to your XML sitemap.

Firewalls can block bots even when robots.txt allows them

A WAF or CDN blocks requests before a bot ever reads robots.txt. Cloudflare’s AI Crawler and Agent access policies (Security, Settings) can set Block for the Search, Agent and Training groups (Cloudflare Docs, accessed November 2026). Setting Training to Disallow with Bot Preference Sync makes Cloudflare prepend Disallow lines for GPTBot, ClaudeBot, Google-Extended and others to your robots.txt (Cloudflare Docs, managed robots.txt). The AI Search Checker cannot see this layer, so check your server logs: bot requests answered with 403 are being blocked.

Frequently asked questions about robots.txt

What if my site has no robots.txt?

The AI Search Checker treats every bot as allowed and gives the full 20 points. It is still worth creating one to declare your sitemap.

How is robots.txt different from llms.txt?

Robots.txt allows or blocks bots. llms.txt only suggests pages worth reading, and no major vendor has confirmed using it.

If you want your robots.txt reviewed for AI search, see Adtek’s SEO solution.