Robots.txt has a root block for supported AI crawler tokens
ShipReady found a root Disallow directive in robots.txt that applies to one or more supported AI crawler tokens under its bounded parser.
Why it matters
A robots.txt rule can be an intentional content-access policy. This finding identifies a parsed root rule for ShipReady's supported AI crawler tokens without determining whether a particular assistant visited, read, cited, or used the site.
How ShipReady detects it
What it inspects
ShipReady makes a bounded, redirect-following request for /robots.txt and parses its User-agent, Allow, and Disallow groups after removing comments.
What triggers a finding
The rule triggers when a root Disallow: / applies to a supported AI crawler token: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, or CCBot. Named agent groups take precedence over wildcard groups, and an explicit root Allow suppresses the block.
When Googlebot or Bingbot is also blocked, ShipReady reports only AI crawlers explicitly named in robots.txt, so a wildcard search-engine block is not repeated as a separate AI finding.
What the evidence shows
The fetched robots.txt contained a parsed root-disallow instruction applying to the reported supported AI crawler token or tokens.
Limits of this check
ShipReady does not contact the listed crawler services, verify crawler compliance, determine whether an assistant uses a token, crawl the site as each service, or establish citation, training, search, or legal outcomes. Missing, unavailable, unsafe, or non-200 robots responses do not create this finding.
How to fix it
Confirm the intended AI-crawler policy
Review whether each reported crawler block is a deliberate content-access decision for this origin before changing robots.txt.
Adjust only the intended crawler directives
If a supported crawler should not be root-blocked, update the applicable User-agent group and verify that a more-specific group does not override the intended policy.
How to verify the change
Verify the deployed robots directive
Fetch the deployed robots.txt and inspect the effective group for each supported crawler relevant to the deployment.
Expected: The root allow/disallow directive for each intended supported AI crawler matches the chosen policy.