Your CDN blocks AI crawlers before they reach your site

Why it matters

Requesting your homepage as an ordinary browser returns {status_code}, but requesting the same URL as GPTBot returns 403. Something in front of your application — a CDN bot-protection rule, a WAF, or a hosting-level setting — is rejecting AI crawlers by user-agent. This is worse than a robots.txt rule because it is invisible: your robots.txt can be perfectly permissive and you would still be unreachable, with nothing in your own configuration to explain why ChatGPT and Perplexity never mention you.

How ShipReady detects it

Whether AI assistants can see the site at all. Two distinct failures, both invisible from inside the company: 1. robots.txt disallows the AI crawlers. Usually a leftover from the 2023-24 "block the scrapers" period, or a rule copied from a publisher's robots.txt into a SaaS marketing site where it makes no sense. 2. The CDN or WAF rejects them before robots.txt is ever read. This is the worse one, because the site owner can look at a perfectly permissive robots.txt and conclude everything is fine while Cloudflare returns 403 to every AI user-agent at the edge. Neither is a security issue — this pillar sits in the READINESS group, so a finding here moves the Launch Readiness score and leaves Production Risk untouched. That separation is the point: "nobody can find you" and "your database is public" are different problems and should not share a number. Deliberately NOT checked: llms.txt. Independent studies across hundreds of millions of LLM crawler requests show the file is essentially untouched, and Google states it does not influence AI Overviews. Reporting its absence would be advice that does nothing.

Detection is deterministic. ShipReady reports this only when it observes the condition directly, and prefers to miss a real problem over inventing one. Rule version 1.0.0.

How to fix it

This is the prompt ShipReady puts in your report — written to be pasted straight into Cursor, Claude Code, or whichever assistant built the app.

Something in front of your application returns 403 to AI crawler user-agents while serving normal browsers fine, so your robots.txt is never even read. Check your CDN or hosting bot-protection settings: in Cloudflare this is usually the AI Scrapers and Crawlers toggle under Security > Bots, or a custom WAF rule matching on user-agent; Vercel and Netlify have equivalent bot-filtering options. Allow GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended, then re-test by requesting your homepage with one of those user-agents and confirming you get a 200. Control which crawlers you allow in robots.txt, where you can see and change it, rather than invisibly at the edge.

Frequently asked questions

What does "Your CDN blocks AI crawlers before they reach your site" mean?

Requesting your homepage as an ordinary browser returns {status_code}, but requesting the same URL as GPTBot returns 403.

How serious is it?

ShipReady rates this medium. Fix soon. Meaningfully weakens a defence or degrades how the site works.

How do I fix it?

Paste the fix prompt on this page into Cursor, Claude Code or your AI editor. It is the same prompt ShipReady puts in your report.

Can I check my own site?

Yes — ShipReady scans up to ten pages of any public site for free and reports this alongside every other check.

Related checks

• AI assistants are blocked from reading your sitemedium
• Your site tells search engines not to index ithigh