Is Your robots.txt Accidentally Blocking Google?
A single misplaced Disallow directive can wipe your site from search results overnight. Here is how to audit and fix your robots.txt before it becomes a costly problem.
By ShipReady · Updated
robots.txt is four lines of text with the power to remove your entire site from search. It is also one of the most commonly misconfigured files on the web, usually because a staging configuration shipped to production and nobody noticed until traffic fell off a cliff.
How robots.txt actually works
It tells crawlers which paths they may request. It does not remove pages from the index by itself, and it does not secure anything — it is a request, not a lock. This is the exact line that deindexes a site:
User-agent: *
Disallow: /The classic mistakes
- A staging "Disallow: /" shipped to production.
- Blocking /_next, /static or asset folders so Google cannot render your pages.
- Confusing robots.txt with noindex — to remove a page from results, use a noindex meta tag, not Disallow.
- Blocking the AI crawlers (GPTBot, ClaudeBot, PerplexityBot) without meaning to, hurting AI visibility.
Disallow and noindex do opposite things
This is the mistake with the worst consequences, so it is worth being precise. Disallow says "do not fetch this URL". noindex says "do not put this URL in the index". They are not interchangeable, and combining them backwards is self-defeating: if you disallow a page, Google cannot fetch it, so it never sees the noindex you put in the HTML — and the URL can still appear in results as a bare link with no snippet.
The rule: to keep a page out of search, leave it crawlable and mark it noindex. Use Disallow for pages you want crawlers to stop spending budget on — endpoints, authenticated areas, infinite faceted URLs.
| You want | Use | Do not use |
|---|---|---|
| This page out of search results | noindex, page crawlable | Disallow — the noindex is then unreadable |
| Crawlers to stop hitting an endpoint | Disallow | noindex — it still gets fetched |
| A private area protected | Authentication | Either. robots.txt is a request, not a lock, and the file is public. |
How to verify
- Fetch
https://yourdomain.com/robots.txtin a private window and read it with fresh eyes. Confirm it returns 200 andtext/plain, not your app's 404 page rendered as HTML. - Confirm your
Sitemap:line is present and points at a URL that actually resolves. - Check a few important URLs against the rules — Search Console's robots.txt report will tell you which line matched.
- Check the AI crawlers separately:
GPTBot,ClaudeBot,PerplexityBotand friends are often blocked by a wildcard someone added years ago.
The block you cannot see in the file
A robots.txt that allows everything is not proof that crawlers get in. Bot-protection rules at a CDN can return 403 to a crawler before the request reaches your application at all, which is invisible in the file and invisible in your own logs. ShipReady reports that separately as AI crawlers blocked at the edge, because the fix is in a different place entirely.
ShipReady fetches your robots.txt the way a crawler does and reports a missing file, AI crawlers disallowed, a `noindex` shipped to production, and a missing or empty sitemap. If AI visibility is the goal rather than just Google, AEO vs SEO covers what else those systems need.
Frequently asked questions
- Does robots.txt remove a page from Google?
- No. It asks crawlers not to fetch the URL. A disallowed URL can still appear in results as a bare link, because Google can learn about it from other sites. To remove a page from the index, leave it crawlable and serve a `noindex` robots meta tag.
- Is `Disallow: /` the same as taking my site offline?
- For search, effectively yes over time. The site stays reachable to people, but compliant crawlers stop fetching it, existing entries decay, and new pages are never discovered.
- Should I block AI crawlers?
- That is a business decision, not a technical one. Blocking GPTBot, ClaudeBot and PerplexityBot keeps your content out of those systems, and also out of the answers they give about your product. Most small SaaS companies want to be cited; the point is to make the choice deliberately rather than inherit it from a copied file.