AI assistants are blocked from reading your site
Your robots.txt disallows {value} AI crawler{value}, so those assistants cannot read your pages and cannot cite you when someone asks about what you do.
Why it matters
Your robots.txt disallows {value} AI crawler{value}, so those assistants cannot read your pages and cannot cite you when someone asks about what you do. For a product that people discover by asking ChatGPT or Perplexity for recommendations, this removes you from the answer entirely. Rules like these are usually inherited — copied from another site's robots.txt, or left over from a period when blocking AI scrapers was the default advice. If you are blocking them deliberately to protect original content, this is working as intended and you can ignore it.
How ShipReady detects it
Whether AI assistants can see the site at all. Two distinct failures, both invisible from inside the company: 1. robots.txt disallows the AI crawlers. Usually a leftover from the 2023-24 "block the scrapers" period, or a rule copied from a publisher's robots.txt into a SaaS marketing site where it makes no sense. 2. The CDN or WAF rejects them before robots.txt is ever read. This is the worse one, because the site owner can look at a perfectly permissive robots.txt and conclude everything is fine while Cloudflare returns 403 to every AI user-agent at the edge. Neither is a security issue — this pillar sits in the READINESS group, so a finding here moves the Launch Readiness score and leaves Production Risk untouched. That separation is the point: "nobody can find you" and "your database is public" are different problems and should not share a number. Deliberately NOT checked: llms.txt. Independent studies across hundreds of millions of LLM crawler requests show the file is essentially untouched, and Google states it does not influence AI Overviews. Reporting its absence would be advice that does nothing.
Detection is deterministic. ShipReady reports this only when it observes the condition directly, and prefers to miss a real problem over inventing one. Rule version 1.0.0.
How to fix it
This is the prompt ShipReady puts in your report — written to be pasted straight into Cursor, Claude Code, or whichever assistant built the app.
Your robots.txt disallows one or more AI crawlers, so ChatGPT, Claude, Perplexity and Google's AI features cannot read your pages or cite your product. Open your robots.txt and remove or narrow the Disallow rules for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot — most commonly this means deleting a copied block of AI-bot rules, or replacing "Disallow: /" with "Allow: /" for those agents. Keep blocking them only if you have decided you do not want your content used by AI assistants; for a product that people discover by asking an AI for recommendations, that decision removes you from the answer.
Frequently asked questions
- What does "AI assistants are blocked from reading your site" mean?
- Your robots.txt disallows {value} AI crawler{value}, so those assistants cannot read your pages and cannot cite you when someone asks about what you do.
- How serious is it?
- ShipReady rates this medium. Fix soon. Meaningfully weakens a defence or degrades how the site works.
- How do I fix it?
- Paste the fix prompt on this page into Cursor, Claude Code or your AI editor. It is the same prompt ShipReady puts in your report.
- Can I check my own site?
- Yes — ShipReady scans up to ten pages of any public site for free and reports this alongside every other check.
Related checks
Does your site have this?
ShipReady checks this and 73 other things across up to ten pages of your site. Free, no signup.
Scan my site