AI assistants are blocked from reading your site
Your robots.txt disallows one or more AI crawlers, so those assistants cannot read your pages and cannot cite you when someone asks about what you do.
Why it matters
Your robots.txt disallows one or more AI crawlers, so those assistants cannot read your pages and cannot cite you when someone asks about what you do. For a product that people discover by asking ChatGPT or Perplexity for recommendations, this removes you from the answer entirely. Rules like these are usually inherited — copied from another site's robots.txt, or left over from a period when blocking AI scrapers was the default advice. If you are blocking them deliberately to protect original content, this is working as intended and you can ignore it.
How ShipReady detects it
Whether crawlers can see the site at all. Three distinct failures, all invisible from inside the company: 1. robots.txt disallows the SEARCH engines. The catastrophic one: a `Disallow: /` shipped from staging makes the site unfindable in Google and Bing entirely. It is also the quietest, because the site looks perfect to everyone who already has the URL. 2. robots.txt disallows the AI crawlers. Usually a leftover from the 2023-24 "block the scrapers" period, or a rule copied from a publisher's robots.txt into a SaaS marketing site where it makes no sense. 3. The CDN or WAF rejects them before robots.txt is ever read. This is worse than (2), because the site owner can look at a perfectly permissive robots.txt and conclude everything is fine while Cloudflare returns 403 to every AI user-agent at the edge. WHY (1) AND (2) ARE SEPARATE RULES, AND WHY (2) DEFERS TO (1) ──────────────────────────────────────────────────────────── A blanket `User-agent: * / Disallow: /` blocks the AI crawlers too, so before rule (1) existed a site invisible to Google reported exactly one finding: "your robots.txt disallows 7 AI crawlers", at medium severity. That is true, and it names the smaller consequence while staying silent on the larger one — a founder reads it as a visibility nit rather than as "Google cannot index this site". So the AI rule now reports only crawlers blocked by a group that NAMES them. An AI crawler caught by the same wildcard that blocks Google is the same fact stated twice, and the search-engine finding is the one that should be read. A group written specifically as `User-agent: GPTBot` is a separate, deliberate decision and keeps its own finding. None of these is a security issue — this pillar sits in the READINESS group, so a finding here moves the Launch Readiness score and leaves Production Risk untouched. That separation is the point: "nobody can find you" and "your database is public" are different problems and should not share a number. Deliberately NOT checked: llms.txt. Independent studies across hundreds of millions of LLM crawler requests show the file is essentially untouched, and Google states it does not influence AI Overviews. Reporting its absence would be advice that does nothing.
Detection is deterministic. ShipReady reports this only when it observes the condition directly, and prefers to miss a real problem over inventing one. Rule version 1.2.0.
How to fix it
This is the prompt ShipReady puts in your report — written to be pasted straight into Cursor, Claude Code, or whichever assistant built the app.
Your robots.txt disallows one or more AI crawlers, so ChatGPT, Claude, Perplexity and Google's AI features cannot read your pages or cite your product. Open your robots.txt and remove or narrow the Disallow rules for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot — most commonly this means deleting a copied block of AI-bot rules, or replacing "Disallow: /" with "Allow: /" for those agents. Keep blocking them only if you have decided you do not want your content used by AI assistants; for a product that people discover by asking an AI for recommendations, that decision removes you from the answer.
Frequently asked questions
- What does "AI assistants are blocked from reading your site" mean?
- Your robots.txt disallows one or more AI crawlers, so those assistants cannot read your pages and cannot cite you when someone asks about what you do.
- How serious is it?
- ShipReady rates this medium. Fix soon. Meaningfully weakens a defence or degrades how the site works.
- How do I fix it?
- Paste the fix prompt on this page into Cursor, Claude Code or your AI editor. It is the same prompt ShipReady puts in your report.
- Can I check my own site?
- Yes — ShipReady scans up to ten pages of any public site for free and reports this alongside every other check. The free report lists every issue it finds and shows full evidence and a fix prompt for the critical and high-severity ones; medium and low findings are counted and unlock on Pro.
Related checks
Run this check on your site
ShipReady checks this and 193 other things across up to ten pages of your site, with an AI-ready fix for each. Free, no signup.