Missing robots.txt
Without robots.txt, crawlers work from defaults: they cannot find your sitemap, and you have no way to keep them out of routes that should not be indexed.
Why it matters
Without robots.txt, crawlers work from defaults: they cannot find your sitemap, and you have no way to keep them out of routes that should not be indexed. It is also the file AI crawlers read first, so it is where your visibility to assistants begins.
How ShipReady detects it
Launch signals that belong to the ORIGIN: whether unknown routes return a real 404, and whether robots.txt and sitemap.xml exist. All three are single facts about the deployment. Re-probing them per page would issue three requests per crawled page to re-learn the same answers.
Detection is deterministic. ShipReady reports this only when it observes the condition directly, and prefers to miss a real problem over inventing one. Rule version 1.2.0.
How to fix it
This is the prompt ShipReady puts in your report — written to be pasted straight into Cursor, Claude Code, or whichever assistant built the app.
A 'robots.txt' file was not found at the root of the domain. Add a robots.txt file to guide search engine crawlers and restrict access to sensitive paths.
Frequently asked questions
- What does "Missing robots.txt" mean?
- Without robots.txt, crawlers work from defaults: they cannot find your sitemap, and you have no way to keep them out of routes that should not be indexed.
- How serious is it?
- ShipReady rates this low. Worth fixing. Small individually, and they accumulate.
- How do I fix it?
- Paste the fix prompt on this page into Cursor, Claude Code or your AI editor. It is the same prompt ShipReady puts in your report.
- Can I check my own site?
- Yes — ShipReady scans up to ten pages of any public site for free and reports this alongside every other check.
Related checks
Does your site have this?
ShipReady checks this and 73 other things across up to ten pages of your site. Free, no signup.
Scan my site