Robots.txt probe did not return HTTP 200
ShipReady's answered, redirect-following probe of /robots.txt did not return HTTP 200.
Why it matters
A robots.txt file can be useful for publishing crawler instructions and sitemap locations. This finding reports one bounded response condition; it does not determine a crawler's behavior or whether the origin is indexable.
How ShipReady detects it
What it inspects
After a successfully fetched content-serving page, ShipReady makes one redirect-following request to the origin's /robots.txt path.
What triggers a finding
The rule triggers when that answered robots.txt request returns a status other than 200.
What the evidence shows
ShipReady's /robots.txt probe received a non-200 response after redirects.
Limits of this check
Network failures and rate-limit or unavailable responses are treated as inconclusive and do not trigger. ShipReady does not inspect every hostname or route, parse a non-200 body, verify crawler behavior, or establish search indexing, access control, or assistant visibility.
How to fix it
Confirm whether robots.txt is needed
Review the origin's intended crawler and sitemap-discovery configuration before adding or changing a file.
Serve the intended robots.txt response
If the origin should publish robots.txt, configure the application, proxy, or CDN to serve the intended file at the root path.
How to verify the change
Inspect the deployed robots.txt response
Request /robots.txt on the production origin and inspect its final status and contents.
Expected: The intended robots.txt path returns the expected deployed file.