Robots.txt has a root block for a supported search crawler
ShipReady found a root Disallow directive in the fetched robots.txt that applies to Googlebot or Bingbot under its bounded robots parser.
Why it matters
Robots.txt communicates crawler instructions. A root block for a supported search crawler is a configuration signal to review for a public site, but a parsed instruction alone does not establish a search engine's current crawl or index state.
How ShipReady detects it
What it inspects
ShipReady performs a bounded, redirect-following request for /robots.txt on the scanned origin and parses User-agent, Allow, and Disallow groups after removing comments.
What triggers a finding
The rule triggers when the parser finds a root Disallow: / applying to Googlebot or Bingbot. A named crawler group takes precedence over a wildcard group, and an explicit root Allow suppresses the block for that crawler.
What the evidence shows
The fetched robots.txt contained a supported parsed root-disallow instruction applying to one or both supported search crawler tokens.
Limits of this check
ShipReady does not query Google or Bing, crawl every URL, inspect every crawler, or establish whether pages are indexed, removed, ranked, or received the instruction. Missing, unavailable, unsafe, or non-200 robots responses do not create this finding; the robots retrieval is bounded.
How to fix it
Confirm the origin's intended search visibility
Review whether this is a public production site, staging environment, internal tool, or intentionally private surface before changing robots.txt.
Adjust the intended crawler directives
For a public site that should allow search crawling, remove or narrow the root block for the affected supported crawler group and review related noindex configuration separately.
How to verify the change
Inspect the deployed robots.txt
Fetch the deployed robots.txt and confirm the Googlebot and Bingbot groups have the intended root allow/disallow behavior.
Expected: The parsed directive for each intended supported search crawler matches the deployment's search-visibility decision.