AI Crawlers Robots.txt: What a 122-Site Check Actually Shows About Blocking and Visibility

An October 19, 2026 Reddit post checked robots.txt rules on 122 recognisable websites against 19 AI-related user agents. Its most useful reported observation was not a universal blocking rate, but a specific split: six named news publishers blocked ClaudeBot while allowing GPTBot.

For teams reviewing AI crawlers robots.txt policies, the concrete payoff is avoiding a costly category error: a Disallow directive can affect whether a compliant bot fetches a URL, but it does not by itself prove whether a page will rank, be indexed, train a model, receive a citation, or appear in an AI-generated answer. We need to evaluate each of those outcomes separately.

What the 122-site check did—and did not—establish

The original material is an informal Reddit submission, not a representative academic, industry, or vendor-backed survey. The author described resolving robots.txt policies across 122 well-known sites for 19 AI-related crawlers, using crawler-specific rules, wildcard fallback rules, and Allow/Disallow path matching.

That makes the post a useful policy snapshot from October 2026. It does not establish a reliable web-wide percentage of sites that block AI bots, nor does it prove why any individual company made its decision. The site list was selected rather than randomly sampled, rules can change without notice, and robots.txt does not disclose internal commercial arrangements.

The source did report several concrete observations:

• Six news publishers—The Washington Post, The Guardian, The Verge, The Wall Street Journal, Wired, and Ars Technica—were observed blocking ClaudeBot while allowing GPTBot.
• Khan Academy was described as showing the reverse combination: GPTBot blocked and ClaudeBot allowed.
• Bytespider was reported as the crawler most often blocked within the checker’s selected set, ahead of GPTBot and ClaudeBot.
• The author also reported checking for llms.txt files and filtering apparent soft 404s: URLs that returned HTTP 200 but served an error page rather than a genuine file.

Those observations are worth auditing on the live sites before relying on them operationally. We should describe them as reported examples from one checker, not as conclusive market statistics.

AI crawlers robots.txt patterns reported in the snapshot

The reported examples show that “block AI” is rarely a single policy. A site can permit one named user agent, restrict another, allow the homepage but restrict a content directory, or rely on a broad wildcard group.

The table deliberately avoids claims such as “social networks block nearly all bots” or “publishers are the most restrictive industry.” The source material does not provide identified sites and counts sufficient to support those stronger conclusions.

For background on the different systems that can affect AI retrieval, see our guide to how LLMs index the web.

Why the ClaudeBot and GPTBot split matters

The six-publisher ClaudeBot/GPTBot split is meaningful because it demonstrates that AI crawler policies can be vendor and agent specific. It is not evidence that one provider has a licensing agreement, better citation terms, or privileged access to those publishers.

We should also avoid treating agent names as interchangeable. OpenAI documents GPTBot and OAI-SearchBot separately in its crawler documentation, with separate user-agent controls. Googlebot is Google Search’s crawler and is governed by the crawling and indexing guidance in Google Search Central. ClaudeBot is a named Anthropic-associated crawler included in the informal checker, but a robots.txt line alone does not tell us every downstream product use associated with content that a provider may access.

The possible explanations for the reported publisher split remain unknown. They could include:

1. A deliberate provider-specific business policy.
2. A historical directive that has not been reviewed recently.
3. A technical template, CDN setting, or legal review process.
4. A distinction the publisher attempts to make between different forms of automated access.
5. An operational decision about crawl load or scraping risk.

None of these explanations can be confirmed from robots.txt alone. The practical lesson is narrower: every named AI user agent should have an owner, a documented purpose, and a review date.

Crawling, indexing, training, and citations are different questions

A robots.txt rule is primarily a crawl-access instruction for compliant crawlers. For example:

User-agent: ClaudeBot
Disallow: /

This asks that crawler not to fetch paths on the site. It does not create a universal instruction that every AI model, search engine, dataset, cache, or third-party publisher must forget the content.

For AI visibility work, we separate four layers:

• Crawling: Can a specific bot fetch a URL now?
• Indexing: Can a search engine store and surface a URL in its results?
• Training or dataset use: Has content been collected or used in a model-development process? A current robots.txt rule cannot reliably answer what happened in the past.
• Answer-time citation or mention: Does an AI product cite, link to, or name the site or brand for a specific prompt?

A brand can be mentioned even when its own site is unavailable to one crawler. An assistant may rely on historical information, a search index, a review site, a distributor, a news article, or another third-party source. Conversely, allowing a crawler does not guarantee fetching, indexing, citation, recommendation, referral traffic, or a favourable answer.

That is why crawler access is a diagnostic input rather than an AI visibility metric. We measure what ChatGPT, Claude, Gemini, Perplexity, and Grok actually say in buyer prompts, then compare the brand, citations, and competitors that appear.

Robots.txt versus noindex versus real access control

The most consequential robots.txt issues come from using the wrong mechanism for the desired outcome. robots.txt, noindex, and authentication operate at different layers.

Robots.txt controls crawler access

A bot-specific Disallow directive expresses a preference that a compliant crawler not request matching URLs. RFC 9309 describes the Robots Exclusion Protocol and its rule-matching framework. It is useful for managing crawler access, but it is not a security mechanism.

A broad rule such as the following requires particular care:

User-agent: *
Disallow: /

It can affect Googlebot alongside AI crawlers, depending on more specific groups elsewhere in the file. Teams should not paste generic AI-blocking templates without testing the exact user-agent groups and paths they affect.

Noindex controls search indexing

A crawlable page can carry a robots meta tag such as:

<meta name="robots" content="noindex">

Google’s noindex guidance explains a key technical dependency: Googlebot needs to crawl the page to see the tag. If robots.txt prevents Googlebot from crawling it, Google may still discover the URL from links but cannot process the page-level noindex instruction in the normal way.

Authentication and removal restrict access

For confidential documents, paid content, customer data, or legally restricted material, use authentication, removal, permission controls, or an appropriate server response. Robots.txt is not a barrier against people or non-compliant scrapers.

The distinction matters because a team that blocks Googlebot while adding noindex may accidentally make its indexing objective harder to verify. Google Search Console’s URL Inspection and robots.txt tools are the appropriate checks for Google-specific outcomes.

How to audit AI crawler rules without overreading them

A useful audit begins with the live response at https://example.com/robots.txt. CMS previews, copied policy documents, and old tickets are not substitutes for what the server returns to a crawler.

Use this six-step process:

1. Fetch the live file. Save the HTTP status, response headers, body, retrieval time, and final URL after redirects.
2. List every relevant user agent. Include ClaudeBot, GPTBot, OAI-SearchBot, Googlebot, Bytespider, and any bot that appears in your verified server logs.
3. Resolve path-level rules. Test /, core product pages, documentation, editorial articles, media files, and any restricted directories such as /account/ or /api/.
4. Check wildcard interactions. A specific user-agent group and User-agent: * can produce different results. Do not assume that a single visible Disallow line applies to every bot.
5. Verify Google separately. Use Google Search Console for Googlebot crawlability, indexing status, and live URL inspection rather than inferring Google outcomes from an AI-bot rule.
6. Review logs and answer outputs. Origin logs show attempted requests; AI visibility tracking shows whether answers mention or cite the brand. Neither replaces the other.

The llms.txt soft-404 observation from the Reddit post is especially useful here. HTTP 200 alone is not evidence that a meaningful file exists. Inspect the response body, content type, and whether the file contains intentional content rather than a branded error template.

Our AI crawler server-log analysis explains why observed requests and declared rules should be reviewed together.

Should publishers block AI crawlers for training or citations?

There is no universal answer because “AI crawler” combines several distinct use cases. A publisher may want to limit crawl load, reduce automated reuse of paid reporting, protect user-generated material, or wait for clearer commercial terms. Those are legitimate reasons to evaluate a block.

Equally, a publisher may decide that allowing a particular crawler is compatible with its goals. It may hope to support accurate retrieval of current public information, visibility in source-linked answers, or discovery through AI search products. Those are possible outcomes, not guaranteed benefits: permission alone does not create attributable traffic, improve citation frequency, or ensure answer accuracy.

We recommend making the decision per agent and content type. A blanket block may be right for a premium newsroom, but unnecessary for a public documentation centre where accurate access could be useful. The expected outcome should be written as a hypothesis and tested, not assumed.

What brands should measure after a policy change

For brands, the key business question is usually not “Do we allow GPTBot?” It is “When a buyer asks a high-intent question, are we named, cited, and compared favourably with competitors?”

A technical change can be a reasonable hypothesis when a visibility trend changes, but correlation is not causation. If a product documentation directory is blocked from a crawler, weaker direct citations might be possible; a content refresh, product change, competing sources, model update, or prompt wording could also explain the result.

We would track at least four measures before and after a meaningful access-policy change:

• Brand mention rate across a stable prompt set.
• Citation or source-link presence where an engine provides citations.
• Competitor share of answer for the same buyer questions.
• Crawl activity in verified logs for the relevant named agent.

Run the same prompts across ChatGPT, Claude, Gemini, Perplexity, and Grok rather than treating one engine as a proxy for all. A ClaudeBot directive is not evidence of every Claude answer pathway, just as a GPTBot directive is not a universal ChatGPT visibility switch.

Our field-test method for local AI visibility tracking provides a practical structure for building stable, intent-led prompt sets.

A policy checklist for SEO and content teams

Before changing a robots.txt rule, document the decision with specific operational details. This avoids an unexplained block being inherited for years after its original purpose has disappeared.

1. Name the exact user agent. “AI crawler” is too broad for an implementable policy.
2. State the intended use case. Is the concern training-related collection, answer-time retrieval, search crawling, server load, or abusive scraping?
3. Define the affected URL groups. Product pages, docs, newsroom articles, downloads, and account areas may need different rules.
4. Check conventional SEO risk. Test Googlebot access and noindex interactions in Google Search Console.
5. Record the date, owner, and evidence. The October 2026 snapshot shows why policies need context; the directive itself does not explain its rationale.
6. Set a review date. Recheck vendor documentation, server logs, and actual AI-answer outcomes after a defined period.

For brands using our desktop tracker, that final step means comparing prompt-level visibility before and after the change with the same prompts and the same engines. It turns a robots.txt decision from a theory about access into an observable visibility experiment.

FAQ

Which AI crawlers do websites commonly block in robots.txt?

The informal 122-site checker reported Bytespider as the most frequently blocked agent in its selected set, followed by GPTBot and ClaudeBot. That is not a web-wide ranking: the post used 122 selected sites and 19 named agents at one point in October 2026. Check your own robots.txt, logs, and business objectives before copying another site’s policy.

How many major websites block ClaudeBot, and why?

The source specifically identified six news publishers—The Washington Post, The Guardian, The Verge, The Wall Street Journal, Wired, and Ars Technica—as blocking ClaudeBot while allowing GPTBot. It did not establish why they did so. Robots.txt cannot reveal whether the motivation was commercial, legal, technical, editorial, or simply a historical configuration choice.

Does blocking an AI crawler prevent a page from appearing in search results or AI answers?

No. A robots.txt directive asks a compliant crawler not to fetch matching URLs. It does not automatically remove a URL from Google Search, erase historical model knowledge, prevent third-party discussion, or stop an AI assistant naming a brand. Google indexing and AI-answer citations depend on separate systems, sources, and product behaviours.

What is the difference between robots.txt, noindex, and blocking AI crawler access?

Robots.txt manages crawl access by user agent and path. A noindex tag requests that supporting search engines exclude a crawlable page from their indexes. Authentication or removal restricts actual public access. Googlebot must be able to crawl a page to process its noindex tag, so blocking Googlebot in robots.txt can conflict with an indexing-removal goal.

Should publishers block AI crawlers that use content for training or citations?

Publishers should decide by crawler, content type, and intended use rather than applying one blanket rule. A block may support premium-content, resource, or reuse objectives. Allowing a retrieval-oriented crawler may support current-source access or discoverability, but those benefits remain hypotheses until measured through logs, citations, referrals, and actual answer outputs.