Original research · re-measured 2026-08-14
How often do US small businesses block AI crawlers?
7.9% — 95% range 5.1–11.9%. That is 19 of 242 reachable sites from a random draw of 300, refusing a live-answer AI crawler's exact published user-agent while serving a normal request from the same address. The figure this category usually quotes is 30%, which sits outside our interval — whoever sells you on it is overstating the problem roughly threefold.
The numbers
| Of 300 domains drawn | n | rate | 95% CI |
|---|---|---|---|
| did not serve our fetcher at all | 58 | 19.3% | — |
| reachable, therefore comparable | 242 | — | — |
| robots.txt blocks a live-answer crawler | 3 | 1.2% | 0.4–3.6% |
| refuses one at the firewall | 19 | 7.9% | 5.1–11.9% |
| either — the headline | 19 | 7.9% | 5.1–11.9% |
In this sample every robots.txt blocker also blocks at the firewall, which is why the firewall row and the headline match. The practical consequence: an audit that only reads robots.txt — which is most of them — would have caught 3 of these 19 sites. The check that finds the rest is requesting the page as each crawler and comparing against a control, which is what we run.
“Did not serve our fetcher” is worded that carefully on purpose: 6 of those domains answered a different HTTP client normally, so that bucket mixes sites that are down with sites that refuse scripts, and we cannot always tell which from outside.
The number moved four times, and only ever against us
Every published correction made our headline smaller. We consider that the strongest evidence on this page — an instrument whose errors all point the same way is usually selling something, and ours kept getting stricter instead.
| 10.4% | first sweep | superseded: https-only fetch, timeouts as blocks, no parked detection, no bot-filtering control |
| 9.1% | corrected fetch, wrong claim | superseded: counted 2 domains that block only bingbot and Googlebot |
| 8.3% | AI-only claim, 20 of 241 | superseded: measured before the probe bracketed its controls, so a mid-probe rate limit could still read as a denylist |
| 7.9% | bracketed re-sweep, 19 of 242 | dropped one domain whose all-four-agents block was the rate-limit artifact the bracketing exists to catch (thefancysalon.com) |
How we bound our own false positives
A site could in principle refuse our probe while allowing the real crawler — verified-bot filtering would look exactly like a block to any header-only method. So all 20 domains the pre-correction sweep reported were re-probed from one address with three user-agents: the operator's exact string, a nonsense string containing the same token, and the control. 16 of 20 reproducible blocks are provably string-matching — a rule that refuses zzz-OAI-SearchBot/1.0-qqq is matching the token, and the genuine crawler sends the token — and 0 showed the verified-bot case that would make us wrong.
The same re-probe produced our own error, which we kept: it first read 4 of 20 blockers as having stopped within a day. The true count was 1 — the re-probe was live-HTTP only, and a robots.txt block produces no refusal a live probe can see; it also carried one user-agent, and one domain blocks a different one. An instrument that cannot detect the thing it is refuting is the exact error this page exists to avoid, and we committed it ourselves before catching it.
What this cannot tell you
- Whether a given block is still there. Blocking is a firewall toggle; a block confirmed on Monday can be gone Tuesday, and sites start blocking as easily as they stop. Findings are perishable, which is why we re-verify before any claim reaches a specific business — and why no domain list appears below.
- The verified-bot residual, with certainty. Zero observed in 20 re-probed domains does not prove the class is empty; a larger sample would eventually find one. What it does mean: on the sites we actually report, we could not produce a single case.
- The future. Web Bot Auth (RFC 9421) verifies crawlers cryptographically and is unforgeable by construction, so this method is most valid today and less valid every year as adoption grows. We publish that shelf life rather than pretend it away.
- Who is doing the blocking on purpose. Reported spoofing of AI-crawler user-agents runs around 5.7% of such traffic (secondary source; not independently confirmed — their page refused our fetcher, which we note without comment). From a site's perspective our probe is indistinguishable from that traffic. We are part of the reason verification exists.
What we looked for and do not claim
The first segment analysis suggested service trades block more than the sample average, at p = 0.0096. It was the first result here to flatter the product, so we tested it on an independent 60-business sample before publishing: z = 0.08, p = 0.937 — no difference at all, and the per-vertical rank order inverted between samples. The replication is underpowered (27% power), so the finding is not refuted; it is not established, and no vertical claim appears on this site. We mention it because leaving it out entirely would make the study look cleaner than it is.
Why there is no domain list
Naming a business for a state it may have fixed yesterday is the kind of claim we refuse everywhere else, so we refuse it here. Aggregates only. If you are a researcher and want to reproduce the draw, the frame is a hash-stable OpenStreetMap draw, US small businesses with websites — write to hello@citationchecks.com and we will describe the method to whatever depth you need.
Method in brief, and citing this
300 domains, hash-stable OpenStreetMap draw, US small businesses with websites. Each fetched across https, www and plain http; robots.txt parsed against each live-answer assistant's published crawler token; each site then requested with the operator's exact published user-agent string, bracketed between control requests from the same address so a mid-probe rate limit cannot read as a block. The wider instrument is described on the methodology page.
Cite it as: Citation Checks, “AI crawler blocking among US small businesses” (2026-08-14), https://citationchecks.com/research/ai-crawler-blocking. Free to cite with attribution; if you quote a figure, quote its range too.