Missing robots.txt
Without robots.txt, crawlers work from defaults: they cannot find your sitemap, and you have no way to keep them out of routes that should not be indexed.
Why it matters
Without robots.txt, crawlers work from defaults: they cannot find your sitemap, and you have no way to keep them out of routes that should not be indexed. It is also the file AI crawlers read first, so it is where your visibility to assistants begins.
How ShipReady detects it
Launch signals that belong to the ORIGIN: whether unknown routes return a real 404, whether robots.txt and sitemap.xml exist, and whether the site has a browser-tab icon. All are single facts about the deployment. Re-probing them per page would issue the same requests once per crawled page to re-learn the same answers. The favicon probe is CONDITIONAL: it is only sent when the homepage declares no icon <link> at all, because a declared icon already answers the question and browsers only fall back to /favicon.ico when nothing is declared. On a site with an icon this check costs nothing extra.
Detection is deterministic. ShipReady reports this only when it observes the condition directly, and prefers to miss a real problem over inventing one. Rule version 1.10.0.
How to fix it
This is the prompt ShipReady puts in your report — written to be pasted straight into Cursor, Claude Code, or whichever assistant built the app.
A 'robots.txt' file was not found at the root of the domain. Add a robots.txt file to guide search engine crawlers and restrict access to sensitive paths.
Frequently asked questions
- What does "Missing robots.txt" mean?
- Without robots.txt, crawlers work from defaults: they cannot find your sitemap, and you have no way to keep them out of routes that should not be indexed.
- How serious is it?
- ShipReady rates this low. Worth fixing. Small individually, and they accumulate.
- How do I fix it?
- Paste the fix prompt on this page into Cursor, Claude Code or your AI editor. It is the same prompt ShipReady puts in your report.
- Can I check my own site?
- Yes — ShipReady scans up to ten pages of any public site for free and reports this alongside every other check. The free report lists every issue it finds and shows full evidence and a fix prompt for the critical and high-severity ones; medium and low findings are counted and unlock on Pro.
Related checks
Run this check on your site
ShipReady checks this and 193 other things across up to ten pages of your site, with an AI-ready fix for each. Free, no signup.