Your robots.txt tells search engines to stay out of the whole site
robots.txt contains a root `Disallow: /` that applies to Google and Bing, so they are instructed not to crawl any page on this site.
Why it matters
robots.txt contains a root `Disallow: /` that applies to Google and Bing, so they are instructed not to crawl any page on this site. Pages already indexed drop out over time, and new ones are never picked up. This is the most consequential line a site can ship by accident, and the easiest to miss: everything works for anyone who already has the URL. The usual cause is a staging or preview robots.txt promoted to production, where blocking crawlers was correct and nobody changed it afterwards. Check the file before changing anything. If this site is meant to be found, remove the root `Disallow: /` and allow crawling — then expect indexing to take days to weeks to recover, because it is not immediate. If this is an internal tool, a staging environment or a logged-in app surface that should not be in search, the file is correct as it stands and you can ignore this.
How ShipReady detects it
Whether crawlers can see the site at all. Three distinct failures, all invisible from inside the company: 1. robots.txt disallows the SEARCH engines. The catastrophic one: a `Disallow: /` shipped from staging makes the site unfindable in Google and Bing entirely. It is also the quietest, because the site looks perfect to everyone who already has the URL. 2. robots.txt disallows the AI crawlers. Usually a leftover from the 2023-24 "block the scrapers" period, or a rule copied from a publisher's robots.txt into a SaaS marketing site where it makes no sense. 3. The CDN or WAF rejects them before robots.txt is ever read. This is worse than (2), because the site owner can look at a perfectly permissive robots.txt and conclude everything is fine while Cloudflare returns 403 to every AI user-agent at the edge. WHY (1) AND (2) ARE SEPARATE RULES, AND WHY (2) DEFERS TO (1) ──────────────────────────────────────────────────────────── A blanket `User-agent: * / Disallow: /` blocks the AI crawlers too, so before rule (1) existed a site invisible to Google reported exactly one finding: "your robots.txt disallows 7 AI crawlers", at medium severity. That is true, and it names the smaller consequence while staying silent on the larger one — a founder reads it as a visibility nit rather than as "Google cannot index this site". So the AI rule now reports only crawlers blocked by a group that NAMES them. An AI crawler caught by the same wildcard that blocks Google is the same fact stated twice, and the search-engine finding is the one that should be read. A group written specifically as `User-agent: GPTBot` is a separate, deliberate decision and keeps its own finding. None of these is a security issue — this pillar sits in the READINESS group, so a finding here moves the Launch Readiness score and leaves Production Risk untouched. That separation is the point: "nobody can find you" and "your database is public" are different problems and should not share a number. Deliberately NOT checked: llms.txt. Independent studies across hundreds of millions of LLM crawler requests show the file is essentially untouched, and Google states it does not influence AI Overviews. Reporting its absence would be advice that does nothing.
Detection is deterministic. ShipReady reports this only when it observes the condition directly, and prefers to miss a real problem over inventing one. Rule version 1.2.0.
How to fix it
This is the prompt ShipReady puts in your report — written to be pasted straight into Cursor, Claude Code, or whichever assistant built the app.
Your robots.txt contains a root `Disallow: /`, which tells search engines not to crawl any page on the site. Before changing anything, decide whether that is wrong — for a staging environment, an internal tool or a logged-in app surface it is correct.
IF THE SITE SHOULD BE FOUND, open `/robots.txt` and remove the root disallow. A minimal correct file is:
User-agent: *
Allow: /
Sitemap: https://your-domain.com/sitemap.xml
Keep any narrower rules you actually want (`Disallow: /admin/`, `Disallow: /api/`). It is only the bare `Disallow: /` that removes the whole site.
FIND OUT WHERE THE FILE COMES FROM, or it will come back. Most frameworks generate it: `public/robots.txt` in Next.js, Vite and CRA; `app/robots.ts` in the Next.js App Router; `static/robots.txt` in SvelteKit and Astro. If the file is generated per environment, check that the production branch is not inheriting the preview value — that is the usual cause.
THEN CHECK FOR THE SECOND SWITCH. A `noindex` in a meta robots tag or an `X-Robots-Tag` response header removes pages from search independently of robots.txt, and the two are usually shipped together by the same staging config. Fixing one and not the other leaves the site just as invisible.
AFTERWARDS, verify rather than assume. Fetch the file yourself (`curl https://your-domain.com/robots.txt`), then use Google Search Console → URL Inspection on a real page to confirm it reports the page as crawlable, and submit your sitemap. Recovery is not instant: re-crawling and re-indexing take days to weeks, and nothing you do makes that immediate.
ONE THING NOT TO DO: robots.txt is not an access control. It asks well-behaved crawlers not to read a path; it does not stop anyone. Never use it to hide an admin route or a sensitive file — publishing the path in a file everyone reads is worse than saying nothing. Protect those with authentication.Frequently asked questions
- What does "Your robots.txt tells search engines to stay out of the whole site" mean?
- robots.txt contains a root `Disallow: /` that applies to Google and Bing, so they are instructed not to crawl any page on this site.
- How serious is it?
- ShipReady rates this high. Fix before launch. A real weakness that an attacker can act on.
- How do I fix it?
- Paste the fix prompt on this page into Cursor, Claude Code or your AI editor. It is the same prompt ShipReady puts in your report.
- Can I check my own site?
- Yes — ShipReady scans up to ten pages of any public site for free and reports this alongside every other check. The free report lists every issue it finds and shows full evidence and a fix prompt for the critical and high-severity ones; medium and low findings are counted and unlock on Pro.
Related checks
Run this check on your site
ShipReady checks this and 193 other things across up to ten pages of your site, with an AI-ready fix for each. Free, no signup.