AI Crawler Traffic: What 18 Days of Server Logs Tell Us to Prioritize

18 days of verified AI crawler traffic produced a blunt result: all 14 observed crawlers fetched ordinary pages, while no verified AI or search crawler fetched llms.txt. The practical payoff is clear: we can spend less time treating “AI-friendly” files as a visibility shortcut and more time making key pages accessible, useful, technically sound, and measurable in the AI answers buyers actually see.

That finding comes from an experiment shared by Pournasserian on Reddit and documented in a longer write-up. Between September 30 and October 17, 2026, the site owner reviewed Cloudflare logs for 14 verified crawlers: 11 AI, AI-search, or assistant crawlers and three conventional search crawlers. It is one small-site case study, not a universal crawl-frequency benchmark—but it is a useful reminder that AI crawler traffic is observable behavior, not a checklist item.

What the 18-day crawler-log test actually measured

The experiment added several files commonly recommended in “AI-friendly website” advice, then checked whether verified crawlers requested them. “Verified” matters here: Cloudflare classified the bot using IP, reverse-DNS, or request-signature signals rather than trusting a self-declared user agent.

Across the 14 verified crawlers, the number that requested each path type at least once was:

• Regular pages: 14 of 14
• Feeds: 10 of 14
• robots.txt: 8 of 14
• Markdown page copies: 7 of 14
• XML sitemap: 5 of 14
• llms.txt: 1 of 14
• llms-full.txt: 1 of 14
• /.well-known/ai-catalog.json: 1 of 14

The core lesson is not that every site needs Markdown copies or that feeds always matter. It is that crawlers demonstrably found the conventional web infrastructure first: HTML pages, crawl directives, feeds, and sitemaps.

The timings also need context. The Markdown copies were available for only eight days, while the AI catalog existed for one day. A request means a bot fetched a URL; it does not prove that the content was indexed, retained, ranked, grounded into an answer, or cited to a user. That distinction is central to any useful AI visibility measurement.

AI crawler traffic is not the same as AI answer visibility

A crawler request can show that a provider reached a URL. It cannot independently tell us whether ChatGPT, Claude, Gemini, Perplexity, or Grok will mention a brand when someone asks a commercial question.

For example, GPTBot requesting a product guide 16 times may be useful operational evidence. But a buyer asking “best employee scheduling software for multi-location restaurants” may still receive an answer that cites competitors, public review sites, or sources the crawler-log report does not expose.

We therefore separate AI visibility into two layers:

1. Access and retrieval readiness: Can crawlers and search systems discover, request, and parse the important content?
2. Answer-level visibility: Does the brand appear, receive a citation, or earn a meaningful share of answer for a defined set of buyer prompts?

Both layers matter, but they answer different questions. Server logs help diagnose technical access and content-discovery assumptions. Prompt-level tracking helps establish whether the market actually sees the brand in generated answers.

That is why our approach is to monitor real prompts across engines alongside technical evidence. A site can have healthy AI crawler traffic and weak answer visibility; equally, it can earn mentions through highly visible third-party sources even when a particular bot rarely visits its own domain.

Why llms.txt should be a low-priority experiment

The source test recorded 75 requests for llms.txt over 17 days, but none came from a verified AI or search crawler. The requests were attributed largely to the site owner’s tools, unidentified clients, Lighthouse, an agent-directory bot, and one Claude-User request made on behalf of a person.

That observation aligns with broader evidence available as of November 6, 2026. Ahrefs analyzed 137,210 domains in May 2026 and reported that 97% of published llms.txt files received zero requests. Of the minority receiving traffic, most requests were bot traffic, while named AI tools represented only a limited share. (ahrefs.com)

Google’s documentation is even more direct for Google Search: it says website owners do not need special machine-readable files, AI text files, markup, or Markdown files to appear in Google Search’s generative experiences, because Google Search does not use them. (developers.google.com)

When publishing llms.txt can still be reasonable

A small, accurate llms.txt file can be acceptable when it costs little to maintain and provides a clean orientation document for people or tools that choose to use it. It may also be useful for an organization testing emerging agent workflows.

But we would not treat it as an AI-citation tactic, ranking factor, or substitute for pages that answer customer questions. The maintenance risk is simple: a neglected file can become another stale, contradictory representation of the business.

A sensible rule is to publish it only if all of the following are true:

• It takes minutes rather than a major engineering project.
• It points to canonical, maintained pages.
• Its claims, products, and URLs can be kept current.
• Its success is not measured by its existence, but by verified requests and answer-level outcomes.

Pages, robots.txt, and sitemaps remain the practical foundation

All 14 verified crawlers in the case study requested normal pages. Eight requested robots.txt, and five requested the sitemap. Those numbers make ordinary technical SEO work the highest-confidence place to start.

The sitemap activity was particularly noticeable: ClaudeBot fetched the sitemap 122 times in 18 days, compared with 19 requests from Googlebot and 10 from GPTBot. We should not generalize those exact counts to another domain, because crawl rates vary by site size, publishing cadence, crawl policy, and bot behavior. Still, it is evidence that a sitemap is more than legacy housekeeping.

For pages that matter commercially, we recommend checking:

• The page returns a stable 200 response and loads meaningful HTML without requiring a user interaction.
• The canonical URL is internally linked from relevant navigation, category, or editorial pages.
• The XML sitemap includes the canonical URL and uses truthful lastmod values tied to material content updates—not a sitewide build timestamp.
• robots.txt does not inadvertently disallow the crawler paths the business wants to permit.
• The copy contains clear facts, product context, evidence, and terminology a buyer would use in a prompt.

Google describes robots.txt primarily as a way to manage crawler access and load; it is not a reliable mechanism for keeping pages out of Google’s results. For exclusion, Google recommends methods such as noindex or access controls, depending on the situation. (developers.google.com)

If AI crawler policy is part of your technical governance, our research on AI crawlers and robots.txt and what secret URLs can—and cannot—prove about robots.txt provides useful context: a declaration file and verified behavior are not the same thing.

Markdown copies are promising, but not proven visibility levers

The most interesting experimental result was the performance of Markdown page copies. The site served a .md version of each page as text/markdown, applied noindex, and referenced it in the HTML head with rel="alternate" and type="text/markdown".

Those copies went live on October 10, 2026. By October 11, seven verified crawlers had started fetching them. Out of 152 non-owner requests, seven verified crawlers accounted for 143: Amazonbot made 52, ExaUser 29, OAI-SearchBot 20, GPTBot 16, Applebot 15, ClaudeBot 6, and Meta-ExternalAgent 4. Googlebot and Bingbot made none during the observation period.

That is an intriguing discovery signal, not proof that Markdown produces more citations. The same information may already be available and understandable in well-structured HTML. Adding another format introduces canonicalization, content-parity, analytics, deployment, and maintenance questions.

How we would test Markdown responsibly

Rather than rolling it out across thousands of URLs, use a controlled test on 10 to 30 high-value evergreen pages. Keep HTML as the canonical public experience, add a closely matched Markdown alternate, and document the launch date.

Then compare at least four things over a meaningful period:

1. Verified requests to HTML versus .md paths by bot identity.
2. Response status, bytes transferred, and any access-control anomalies.
3. Organic crawl and indexing stability for the canonical HTML URLs.
4. Brand mentions and citations for a fixed prompt set before and after the test.

The last measure is the hard one. Without it, a successful fetch can be mistaken for successful AI visibility.

Do not trust user-agent strings by themselves

One of the sharpest findings in the source material was the spoofing volume. The logs contained 113 requests identifying as PerplexityBot, 112 as Kimi-SearchBot, 85 as Bingbot, and 75 as Googlebot that Cloudflare did not verify.

A request header is easy to imitate. If we classify every request with a familiar bot name as a real crawler visit, reporting will overstate activity and can lead to bad decisions—such as blocking a user-agent that was never genuinely associated with the provider.

Cloudflare’s AI Crawl Control is designed to provide visibility into identified AI-service access, and its analytics can be queried programmatically through the GraphQL API. Cloudflare also notes that a user-agent filter can be spoofed, which is exactly why verified detection and corroborating evidence are valuable. (developers.cloudflare.com)

For teams not using Cloudflare, verification should follow each provider’s documented process where available: IP-range checks, reverse DNS followed by forward DNS confirmation, signed-request verification, or log enrichment from a trusted infrastructure provider. Keep a separate category for claimed but unverified traffic rather than mixing it into crawler KPIs.

Build an AI crawler traffic report that can drive decisions

JavaScript analytics usually miss most crawler activity because crawlers generally do not execute a site’s analytics scripts. Server logs, CDN logs, and verified bot tooling are the appropriate starting points.

A compact weekly report does not need 30 dashboards. We would capture the following fields for every verified AI or search crawler request:

Cloudflare’s AI Crawl Control dashboard provides crawler, operator, path, status, and traffic views, while its GraphQL tooling supports custom reporting. Available fields and retention can differ by product plan and configuration, so teams should validate their own account before designing a long-term reporting process. (developers.cloudflare.com)

The goal is not to celebrate crawling volume. A 10,000-request spike to a low-value parameterized URL is an operational problem, not a marketing win. The useful question is whether verified bots can reach the pages that explain what you sell, where you operate, why you are credible, and how you differ from alternatives.

Connect crawl evidence to competitor and citation tracking

Crawl diagnostics tell us if content can be reached. They do not tell us which competitors own the answer when the user asks an evaluative question.

Suppose a payroll platform has five strong feature pages, a current pricing guide, customer stories, and a sitemap that crawlers fetch regularly. That is a healthy foundation. Yet prompts such as “best payroll software for construction companies,” “Gusto alternatives for 50 employees,” or “payroll platform with certified payroll reporting” may still name Gusto, ADP, Paychex, or Rippling instead.

That is where prompt-level visibility measurement matters. We recommend maintaining a fixed prompt set across awareness, comparison, use-case, and purchase-intent stages, then recording for each engine:

• Whether the brand is mentioned.
• Whether it is cited or linked as a source.
• Which competitors appear instead.
• Whether the wording is positive, neutral, qualified, or inaccurate.
• Which source types the answer uses: vendor sites, reviews, publishers, directories, forums, or documentation.

This approach complements technical log analysis rather than replacing it. If no verified crawlers can access a key comparison page, fix access first. If crawlers access it but competitors still dominate answers, investigate the page’s evidence, topical depth, external corroboration, entity clarity, and the sources repeatedly cited by the AI engine.

For a deeper process on turning that insight into content work, see our guide on how to get your brand cited by ChatGPT. And because AI answer visibility sits beside—not instead of—search traffic, it is worth understanding which Google clicks AI Overviews are displacing.

A 30-day implementation plan

The source experiment ran for 18 days, enough to challenge assumptions but not enough to establish permanent crawl behavior. For most brands, a 30-day baseline is a more useful first operating cycle.

Days 1–7: establish the baseline

Inventory priority pages: category pages, solution pages, comparison pages, pricing, locations, original research, and support documentation. Confirm their canonical URLs, sitemap inclusion, HTTP status, crawl directives, and internal links.

Turn on or export bot-aware CDN/server logs. Separate verified AI/search bots, unverified bot-name traffic, ordinary users, monitoring tools, and your own scripts. Record the date when reporting begins.

Days 8–21: fix access and content gaps

Repair obvious 4xx, 5xx, redirect-chain, canonical, and accidental-disallow issues affecting priority URLs. Review lastmod accuracy and verify that the sitemap lists only canonical, indexable pages.

Then improve a small group of pages with missing buyer information: implementation details, constraints, pricing logic, supported integrations, evidence, locations served, or comparison criteria. Do not add formats merely because a trend list says they are “AI-ready.”

Days 22–30: measure outcomes, not just requests

Run a repeatable set of real buyer prompts across the AI engines your customers use. Compare mention rate, citation rate, competitor presence, and share of answer against the initial baseline.

If you test .md alternates or llms.txt, mark the launch date and assess the change separately. A valid outcome is “we observed requests but no answer-visibility improvement.” That result can save far more time than declaring a new file format a best practice without evidence.

What this evidence does—and does not—prove

This study is valuable because it uses verified traffic rather than assumptions. But the site was small and personal, the dataset was Cloudflare-based, and file types were live for different lengths of time. Those constraints mean the crawler mix and request counts may differ considerably on an enterprise publisher, ecommerce store, SaaS site, or local business website.

It also cannot tell us what happens inside private retrieval systems, training pipelines, browser agents, logged-in experiences, or tool calls that do not generate a public crawl. Nor can it establish that crawler requests cause AI citations.

What it does establish is narrower and still important: in this observed period, conventional web assets were the most consistently requested, Markdown alternates were quickly discovered by several verified AI crawlers, and llms.txt did not receive a verified AI/search-crawler fetch. That is enough reason to put measurement ahead of mythology.

FAQ

Does AI crawler traffic mean my brand will be cited in ChatGPT?

No. AI crawler traffic shows that a bot requested a URL; it does not prove the page was indexed, selected for retrieval, trusted, or cited in an answer. Measure both technical access and prompt-level brand mentions. A useful reporting setup combines verified log data with recurring buyer-question tests across the AI engines relevant to your audience.

Should we add an llms.txt file in 2026?

Treat llms.txt as an optional, low-cost experiment—not a core AI visibility tactic. Google says its Search systems do not use special AI files or Markdown to determine visibility in generative Search features, and Ahrefs found 97% of observed llms.txt files received no requests in May 2026. Maintain it only if you can keep it accurate. (developers.google.com)

Are Markdown copies of website pages worth creating?

They may be worth a controlled test, particularly for evergreen documentation or resource pages. In the 18-day experiment, seven verified crawlers fetched Markdown copies shortly after launch. However, there is no demonstrated causal link between Markdown alternates and higher AI citation rates, so preserve canonical HTML and measure answer-level outcomes before a broad rollout.

How can we verify whether an AI crawler is genuine?

Do not rely only on the user-agent string. Use a CDN or server-security product with verified-bot classification, or follow the crawler operator’s published validation procedure, such as IP and reverse-DNS checks where supported. Keep traffic that merely claims to be Googlebot, Bingbot, or an AI crawler in a separate unverified category.

What should we monitor first for AI visibility?

Start with priority-page accessibility, status codes, canonical URLs, robots.txt, sitemap coverage, and verified crawler requests. Then track a stable set of buyer prompts for mentions, citations, competitor inclusion, and answer quality. That sequence prevents a common mistake: optimizing a speculative file format before confirming that core pages are reachable and competitively useful.