Why we compared five websites
If a crawler starts visiting more often, something has become easier or more interesting to collect. That makes the activity worth watching.
The tempting next sentence is that more search visibility must be on the way. Our five-site study gives that idea a useful test instead of letting it become a slogan.
The sites were selected for similar size and a similar period of first Google observations. They share the portfolio's publishing systems, but cover different subjects. That combination lets us compare a working hypothesis across more than our busiest site.
How the five-site comparison keeps its counts comparable
The October 20 export covers July 9 through October 17, 2026. Google Search Console supplies search impressions and clicks. The validated Good Bot measure counts successful content requests using the warehouse's current classification rules.
Requests for assets or datasets are outside that content measure. So are unsuccessful responses and traffic finally classified as scanning. This is a different count from every request made by something using an AI crawler name.
The growth table compares each site's first and last 14-day averages using the study's comparable coverage. Two dates lack comparable Good Bot data. They stay unavailable rather than becoming zero requests.
The full study also compares page timing and logged work. Similar size does not make these sites identical: their subjects and publishing activity differ.
What the different growth rates taught us
Good Bot activity rose on every site in the group. Google impressions rose on AI Symantix and AI to AI Connect, while the other sites moved down in this period comparison.
That is useful because it changes the next decision. A busy log tells us crawlers are reaching content. The search rows tell us whether Google is showing it more often. Neither observation can substitute for the other.
AI Symantix had the strongest impression growth and also accounted for most of the group's logged work. That is a reason to keep page changes and publication dates beside the counts. It is not a reason to assign every gain to crawler traffic or every difference to one tactic.
The comparison leaves crawler activity useful. It shows where collection is happening and which pages may deserve closer attention. It also stops us turning an access signal into an unsupported promise about rankings.
Following the crawler-only pages
The next step is page-specific. We can watch a crawler-only page, note when it first earns a Google impression and keep the dates of useful edits beside it. Then we can compare pages with similar starting conditions.
For AI search itself, the next receipt is an answer. A dated response naming the source page tells us something a crawl count cannot. That is why the Lab keeps AI citations separate from requests.
The live cohort report continues to update. This case preserves its October 20 reading so future results can be compared with what we actually knew at this point.
What we are watching across the five sites next
Follow the same cohort over the next complete reporting window, preserving unavailable dates. Compare page-level crawler timing with first Google impressions and dated content work, then record AI answer citations separately.
The five-site report and its data
- Live report: AI Bot Visibility Across Five Matched Websites
- Live five-site public dataset
- AI Symantix page-level crawler study
- Can AI Bot Data Measure AI Search Visibility?
This is a dated case study from our own portfolio. Public sources and the downloadable snapshot let you check the figures; the Digital Karma Data Warehouse supplies the private search and request records.