68.9 Million AI Crawler Visits Reveal What Drives Visibility in AI Search

Updated on October 19, 2026: figures reviewed and refreshed.

A study based on 68.9 million AI crawler visits shows that websites with a clear, well-organized structure tend to be crawled more by AI (this is a correlation; it does not prove cause and effect).

A new analysis of 858,457 websites hosted on the Duda platform shows how AI crawlers are interacting with websites at scale. The data gives a clearer view of how fast data collection (crawling) is growing, and it also points to the specific things SEO specialists and businesses need to do to increase the traffic they receive from AI search.

AI crawling enters a major growth phase

AI crawling is growing quickly, and the number of crawl requests is now closely tied to delivering real-time answers to users. Notably, most of this crawl traffic comes from a single AI provider.

The research data also reveals a clear pattern in which websites AI crawlers are visiting and, more importantly, why AI gives those particular websites priority when it collects data.

Year-over-year growth in referral traffic from large language models (LLMs)

Referral traffic from LLM platforms rose sharply in the year leading up to the Duda study (published in April 2026). Many platforms posted significant growth, even though they started from very different baselines.

AI referral traffic trends and patterns:

• Total LLM referrals: from 93,484 to 161,469 (+72.7%)
• ChatGPT: from 81,652 to 136,095 (+66.7%)
• Claude: from 106 to 2,488 (23x growth)
• Copilot: from 22 to 9,560 (a sharp rise from almost zero)
• Perplexity: from 11,533 to 13,157 (+14.1%)

The pace of growth was not the same from one platform to the next, but overall, traffic coming from AI systems is rising. The trend shows that users who search for and discover content through AI are becoming an increasingly important source of traffic, rather than just a small supplementary channel.

AI crawlers collect content mainly to deliver answers

AI crawlers are no longer used mainly for indexing purposes. Most of their activity today goes into retrieving content in real time in order to provide answers to users.

Most crawling now happens in response to queries from users rather than to build a search index for a website, and this shift changes both the way content can be accessed and the way that content gets used.

• User Fetch (real-time answers): 56.9% of crawler activity, almost all of it from ChatGPT.
• Training (model training): 28.8%, split between GPTBot and other crawlers.
• Discovery (content indexing): 14.3%, spread across many different systems.
• ChatGPT User Fetch (volume of ChatGPT user retrievals): about 39.8 million visits.

These trends come mainly from ChatGPT, which accounts for nearly all real-time content retrieval. In other words, the move toward crawling for answers is not happening evenly across the market. It is concentrated in one platform that is shaping how content gets accessed. This could change with the arrival of Google’s new Google-Agent crawler.

Market concentration in AI crawling

AI crawling is highly concentrated. OpenAI accounts for the large majority of visits, which reflects ChatGPT’s position as the main tool people rely on to find and retrieve information.

• OpenAI: 55.8 million visits (81.0%)
• Anthropic (Claude): 11.5 million (16.6%)
• Perplexity: 1.3 million (1.8%)
• Google (Gemini): 380,000 (0.6%)

Most AI crawling comes from OpenAI, in line with ChatGPT’s role as the main tool for searching and retrieving information. Claude comes second with a smaller share, which points to a different usage pattern, while the rest of the market makes up only a tiny fraction of total crawling activity.

The scale of AI crawling and what it means

AI crawling already operates at a large scale across the entire web, reaching hundreds of thousands of websites and generating tens of millions of requests every single month.

More than half of the websites in the dataset received at least one visit from an AI crawler, which shows that this activity is not limited to a small group of websites.

• Total websites analyzed: 858,457
• Websites with at least one AI crawler visit: 506,910 (59%)
• Total AI crawler visits (February 2026): 68.9 million

AI crawling is not focused only on prominent or high-traffic websites. It has already become widespread, with a consistent presence across most of the websites that exist today.

How crawling relates to real traffic

Websites that get crawled by AI systems tend to show stronger engagement across a number of different metrics.

What the data shows

1. Websites crawled by AI get more traffic from users than websites that are not crawled.
2. Websites with high traffic are more likely to be crawled.

Websites crawled by AI received significantly more user traffic: 527.7 sessions compared with 164.9 sessions for websites that were not crawled. This does not prove causation, but it shows a clear correlation between a site’s ability to attract users and how often AI systems come back to collect its data.

• Average traffic (crawled vs. not crawled websites): 527.7 vs. 164.9 (3.2x higher)
• Average form submissions: 4.17 vs. 1.57 (2.7x higher)
• Average click-to-call: 8.62 vs. 3.46 (2.5x higher)
• Websites with more than 10,000 sessions: crawl rate of 90.5%

AI systems do not seek out weak or inactive websites to lift them up. Instead, AI keeps returning to websites that already attract users. For marketers, the logic has shifted from trying to “get crawled” to building real demand from users, because visibility in AI systems appears to follow market interest.

Which factors are linked to AI crawler activity?

The study compared websites with third-party integrations, structured data and in-depth content against websites without these elements, to identify which factors had the most influence on AI crawling and AI referrals.

Across the whole dataset, 59% of websites received at least one AI crawler visit in February 2026. The sites crawled most often usually had three groups of signals at the same time: external system integrations, structured business data and in-depth content.

1. External system integrations

These integrations connect a website to outside systems whose job is to verify and distribute information about the business.

• Yext integration: crawl rate of 97.1% vs. about 58% for websites without it (38.9 percentage points higher)
• Review integrations: crawl rate of 89.8% vs. about 58.8%, with an average of 376.9 crawler visits

Websites connected to external data and review systems get crawled more often, which suggests AI uses these signals to decide whether a business is real, verifiable and worth coming back to.

2. Structured websites and business data

The elements below are built directly into the website, and they help AI systems understand and verify the identity of the business.

• Google Business Profile sync: crawl rate of 92.8% vs. 58.9% without GMB sync, with an average of 415.6 crawler visits
• Local schema: crawl rate of 72.3% vs. 55.2% (17.1 percentage points higher); 22.3% of websites in the dataset use local schema
• Dynamic pages: crawl rate of 69.4% vs. 58.2% (11.2 percentage points higher)
• Ecommerce: crawl rate of 54.2% vs. 59.2% for websites that don’t sell online (5.0 percentage points lower)

Websites that make the business identity clear and present information in a machine-readable way tend to be crawled more, which suggests AI favors websites that are easy to understand, easy to verify and easy to extract information from.

3. In-depth content (volume of usable data)

Websites with more content give AI more opportunities to retrieve, reference and reuse information when generating answers. Websites with 50 or more blog posts received an average of 1,373.7 crawler visits, compared with 41.6 for websites with no blog (roughly 33 times more).

Content-rich websites are crawled significantly more often, which suggests AI tends to return to sources that supply a large amount of useful information it can draw on when building answers.

Complete Local Business schema gets more crawling

This part of the study focused on Local Business schema, comparing how completely the schema conveys business information against AI crawling activity. The fields evaluated included business name, phone number, address, opening hours and social media profiles.

• No local schema fields: crawl rate of 55.2%
• 10 to 11 schema fields completed: crawl rate of 82%
• Websites with more complete schema reached a crawl rate 26.8 percentage points higher (82% vs. 55.2%)

Websites that provide fuller local business information as structured data are crawled more often and receive more crawler visits. As the number of completed fields goes up, both crawl rate and crawl frequency rise with it.

The data also shows that when local business information is clearly defined, AI can identify the website more easily. From there, AI is able to verify it and keep coming back to visit. You could call these the prerequisites for getting traffic from AI search.

Key takeaways

AI crawling is becoming a way of discovering content that runs alongside traditional search. The study found clear structural patterns on the websites that crawlers visit most often.

• AI crawling runs in parallel with traditional search and changes how content is accessed and reused.
• Websites with clear local data, in-depth content and complete schema get visited by AI crawlers more often.
• Positive signals usually show up together on the same website rather than in isolation.
• The data shows correlation only and does not prove causation, but the structural patterns on websites need to be consistent.

The data shows that websites which make it easy for AI crawlers to index them and come back tend to perform better. The point worth noting is that a website offering clear, structured, verifiable information, while continuing to build and grow on the back of real demand from users, is more likely to draw AI systems back and benefit from the traffic AI search brings.

This article was adapted by Adtek from the original reporting by Search Engine Journal (2026), updated on October 19, 2026.