Do AI Assistants Respect Robots.txt? What Secret URLs Can—and Cannot—Prove

A Reddit field test reported Grok-related traffic from 59 IP addresses over six requests, frequently using Chrome- or Safari-like user-agent strings. That result is a useful warning for anyone asking do AI assistants respect robots.txt: we can use an unlinked public URL to detect access correlated with a prompt, but we need server-side evidence and careful attribution before claiming an AI provider ignored a rule.

The payoff for SEO teams is a test that can stand up in a client report. Rather than treating one bot log as proof, we can document request timestamps, paths, robots.txt rules, IP verification, user-agent strings, redirects, and the limits of the conclusion. That matters because an answer engine may involve a crawler, a live browser, a search partner, a safety service, or a proxy—not one identifiable “AI bot.”

A secret URL can reveal access that ordinary monitoring misses

The Reddit experiment used a sound core technique: give each assistant an individual, hard-to-guess, unlinked public URL, then ask it to retrieve a code from that page. These were not private pages, authenticated resources, or access-controlled files. They were public URLs deliberately excluded by robots.txt and kept out of links, sitemaps, and other discovery paths.

That distinction is central. If an otherwise undiscoverable URL receives a request immediately after a prompt containing that URL, the event is meaningful. It is evidence that something in the workflow reached the address. It is not automatic proof that the named assistant, its model provider, or one specific crawler made every request.

The reported observations differed by assistant. Claude reportedly requested robots.txt and declined a disallowed URL. ChatGPT reportedly fetched a disallowed URL once, then an OAI-SearchBot request for robots.txt arrived roughly 76 seconds later; on a later attempt it declined without a page request. Perplexity reportedly fetched the homepage and robots file but did not fetch the test URL.

The striking Grok observation was not “59 IPs across six runs.” It was 59 IP addresses over six requests, with browser-like identifiers and infrastructure associated with several network providers. That is enough to justify further investigation. It is not enough to establish a platform-wide xAI policy, deliberate impersonation, or the identity of the operator behind each IP.

Why a robots.txt result is easy to overstate

Robots.txt is a crawl-preference protocol, not an access-control mechanism. RFC 9309 states that the Robots Exclusion Protocol is not authorization. A public URL that needs to remain confidential needs authentication or another actual security boundary; Disallow cannot stop a direct browser request, a human visitor, or an uncooperative client.

With AI systems, the word “assistant” also bundles together multiple possible request types:

1. Training crawlers that collect public material for model-development purposes.
2. Search and indexing crawlers that build a retrieval index.
3. User-triggered fetchers that load a URL because a person asked for it in a chat.
4. Third-party services used for search, rendering, safety review, previews, or proxying.
5. Test contamination, including the tester’s browser, QA tools, uptime monitors, or CDN checks.

OpenAI’s crawler documentation distinguishes GPTBot, OAI-SearchBot, and ChatGPT-User. Google similarly distinguishes common crawlers from user-triggered fetchers in its crawler and fetcher documentation. Those distinctions are why “does ChatGPT respect robots.txt?” is often too broad to be useful.

A better question is: which component requested this path, under which prompt, with what verified identity, and was the event an automatic crawl or a user-directed fetch?

Build an auditable unlinked-URL experiment

A credible test should make accidental discovery unlikely and make later review possible. The most important controls are isolation, complete request logs, and a record of exactly what was prompted. More sophisticated diagnostics are useful, but they are optional—not prerequisites for a meaningful first test.

The essential controls

Create a unique URL for each assistant and condition, such as:

• /ai-test/grok-allow-8f31/
• /ai-test/grok-disallow-8f31/
• /ai-test/claude-allow-93ca/
• /ai-test/claude-disallow-93ca/

Put a different canary phrase on every page, for example violet-maple-8421. Do not include these URLs in page links, XML sitemaps, public documents, screenshots, analytics annotations, or chat transcripts. The Reddit author identified a practical contamination risk after visiting test URLs in their own browser: early log entries reflected their own Brave-on-Windows activity.

For the blocked condition, use a narrow rule that is easy to audit:

User-agent: *
Disallow: /ai-test/grok-disallow-8f31/

Keep a timestamped copy of the exact deployed robots.txt file. If you also add named user-agent rules, record those separately. A test cannot be reproduced if the rule set changes without a record.

At minimum, retain raw server or CDN records containing the UTC timestamp, request method, hostname, path, status code, response size, full user-agent, and source IP or the client-IP header made available by your infrastructure. Also preserve the exact prompt, assistant name, model or mode where displayed, account setting, and session time.

Useful but optional diagnostics

Redirect logs, request IDs, edge location, referrer headers, HTTP version, TLS details, and provider bot scores can all help explain ambiguous events. They are optional because availability varies among Apache, Nginx, Cloudflare, hosting panels, and CDNs. We should not imply that a test is invalid merely because one provider does not expose a bot score or TLS fingerprint.

Likewise, repeating a test can expose inconsistent behavior, caching, or safety decisions, but there is no universal evidence-based number of repetitions. Run enough clearly documented attempts to identify whether an observation is isolated or recurring, and report the actual count rather than presenting “three to five” as a required standard.

Read IP addresses and user agents as evidence, not verdicts

A Chrome-looking user-agent is a clue, not an identity document. Browser automation, remote rendering, proxy networks, security scanners, and ordinary visitors can all send a Chrome or Safari string. Any HTTP client can also spoof a user-agent.

When a request claims to be an official Google crawler, use Google’s published verification process, which describes reverse-DNS and forward-DNS checks. If verification fails, the accurate label is “an unverified request claiming to be Googlebot,” not “Googlebot.”

OpenAI publishes documented identities and controls for its crawlers, but a request arriving after a ChatGPT prompt still needs to be interpreted in context. A named user-agent may be useful corroboration; a browser-like request without a documented identity leaves an attribution gap.

The Grok example shows why this matters. Fifty-nine IPs over six requests could be consistent with distributed browsing or proxy-backed execution. It could also involve a partner, relay, safety layer, unrelated traffic, or a measurement problem. Without vendor-confirmed ranges, a signed request signal, or vendor confirmation, our strongest accurate classification is AI-correlated browser-like traffic.

An IP belonging to AWS, Azure, Google Cloud, Oxylabs, or a residential ISP does not identify the customer using that network. Blocking every browser request from a hosting ASN can also affect legitimate users, accessibility tools, remote employees, and QA systems.

Redirects and canary phrases do not settle attribution

Redirects can create misleadingly simple narratives. If /secret-a returns a 302 to /secret-b, there may be two requests, different IPs, and different services involved. Log the Location header and associate each response with a server-side request ID where possible.

A canary phrase strengthens the access evidence when two things happen close together: the assistant returns the unique phrase and the logs show a request for the matching unlinked public URL. But that combination establishes access correlated with the prompt, not final attribution to a particular assistant component.

A proxy, partner search service, browser execution layer, or safety system may have retrieved the page on the assistant’s behalf. The Reddit experiment’s browser-like and multi-IP findings are precisely why we should avoid writing that a canary plus a log “proves the assistant retrieved it.” It supports a narrower conclusion: the workflow exposed the unique content after the prompt.

We should also avoid reading no request as proof of no access or no model knowledge. The assistant may have declined, used a different source, cached a prior result, lacked browsing in that session, or simply failed to complete the task.

What the reported test suggests about major assistants

The reported observations are useful examples, but they are not a universal ranking of compliant and noncompliant platforms. Model versions, product features, account permissions, regions, and partner services can change behavior over time.

ChatGPT/OpenAI: The report’s initial page fetch followed by an OAI-SearchBot robots request about 76 seconds later is compatible with a multi-component workflow. OpenAI’s documentation distinguishes its crawler identities, so it would be incorrect to conclude from that sequence alone that OAI-SearchBot ignored robots.txt.

Claude/Anthropic: The reported robots request and refusal are consistent with cooperative behavior in that session. They do not establish how every Claude feature, connector, or user-directed browsing action behaves. A single URL and one test date are not a platform policy audit.

Gemini/Google: A plain Google user-agent alone is not enough for attribution. Google documents that common crawlers obey robots.txt for automatic crawling, and it provides a verification procedure for genuine requests. A Gemini prompt, Google Search grounding, Googlebot, and Google-Extended should be recorded as separate test targets where possible.

Perplexity: The report that Perplexity fetched robots.txt and the homepage but declined the unlinked URL as a possible phishing risk may reflect a product safety decision, not a general robots policy. It is evidence about that prompt and session.

Grok/xAI: The browser-style, multi-IP pattern deserves careful logging and vendor clarification. It should not be described as verified xAI crawler traffic until an identity can be independently validated.

Mistral was not part of the reported experiment, so we should not use this test to make detailed claims about Mistral crawler behavior. If we test Mistral separately, it needs its own dated method, logs, and provider documentation.

Practical controls: communicate, limit, or require access

Robots.txt remains useful for communicating crawl preferences to cooperative, named crawlers. It should be part of a layered policy, not a security plan.

For public material that you prefer not to be automatically crawled, deploy specific User-agent directives and retain versioned copies of the file. Review vendor documentation as products change. For expensive, abusive, or unwanted repeated traffic, consider these server-side controls:

• Rate-limit request bursts by IP, session, ASN, or behavior.
• Use WAF rules to log or challenge anomalous patterns before broad blocking.
• Return 401 or 403 where a resource genuinely requires permission.
• Use short-lived signed URLs for temporary assets.
• Put staging sites, client portals, sensitive feeds, and previews behind authentication.

Fail2Ban can help where repeated abusive IP patterns appear in conventional server logs. It is less reliable against rotating IP pools and can create collateral damage when used against browser-like traffic. For material that must not be disclosed, authenticated access is more dependable than attempting to recognize every AI crawler.

Stopping a future fetch also does not erase content already indexed, cached, syndicated, quoted, or included in earlier training data. Our guide to how LLMs index the web explains why crawling, retrieval, indexing, and generated answers need to be measured separately.

Pair access evidence with AI answer evidence

Robots tests answer an operational question: what requests reached our infrastructure under controlled conditions? They do not answer the commercial question: does an AI engine recommend our brand, cite our page, or name a competitor instead?

We recommend keeping two evidence streams:

1. Access evidence: deployed robots rules, verified crawler events, anomalous browser-like requests, redirects, response codes, and authentication outcomes.
2. Answer evidence: real buyer prompts, brand mentions, citations, named competitors, cited URLs, and share of answer by engine.

This distinction prevents inflated reporting. Our GEO reporting guidance for clients covers how to separate observed prompt-level results from broader claims about visibility lift.

AI Visibility Tracker supports the second stream. We built it as a local-first desktop app: teams use their own API keys to run prompt-level checks across ChatGPT, Claude, Gemini, Perplexity, and Grok, then review brand mentions, citations, and competitor gaps. It does not replace raw server logs or claim to identify a browser-like request. Instead, it connects the access question to the outcome that matters: what customers are actually shown in AI-generated answers.

That is increasingly relevant as clicks become less complete as a visibility metric. Our analysis of AI Overviews traffic loss explains why brands need to measure exposure inside the answer as well as referrals after it.

FAQ

Do AI assistants actually respect robots.txt?

Some documented automated crawlers do, but the answer varies by crawler and request type. Google documents robots.txt compliance for common automatic crawlers, while OpenAI distinguishes GPTBot, OAI-SearchBot, and user-triggered actions. A prompt-driven fetch, proxy request, or browser session may not behave like an index crawler. Test the named component and preserve evidence.

Which AI crawlers ignore robots.txt rules?

One field test cannot reliably label an entire AI company as ignoring robots.txt. The reported experiment showed differing behavior among assistants, but browser-style traffic and rotating IPs cannot establish operator identity on their own. Report verified crawler behavior separately from unverified AI-correlated traffic, and avoid turning one session into a platform-wide policy claim.

What does it mean when an AI bot is blocked by robots.txt?

For a cooperative automated crawler, it means the crawler should avoid the disallowed path. It does not make a public URL private, prevent a user-directed browser action, block third-party services, or erase material already collected elsewhere. Use authentication, signed URLs, or network controls when access would create a genuine security or confidentiality risk.

Can AI bots disguise themselves as Chrome or use rotating IP addresses?

Any HTTP client can send a Chrome-like user-agent, and AI retrieval workflows may use browser automation, proxies, or distributed infrastructure. That does not prove deliberate disguise or vendor ownership. Verify named crawlers through the provider’s published method when available, retain raw logs, and classify unverified browser-like traffic conservatively.

How can a website identify whether an AI accessed a secret URL?

Give each assistant a unique, unlinked public URL with a unique canary phrase, then log the path, timestamp, IP, user-agent, status code, and redirects. If the assistant returns the canary and a matching request appears after the prompt, access is strongly correlated. The evidence still may not identify which assistant component, proxy, or partner fetched it.

Is robots.txt still effective for controlling AI crawlers?

Yes, as a documented preference signal for cooperative named crawlers. It is not an access-control system and cannot stop every browser, proxy, or uncooperative client. Combine robots.txt with rate limits, WAF monitoring, and authentication where appropriate, then measure whether permitted content actually appears in relevant AI answers.