GPTBot user-agent probe received 403 while the fetched page answered
ShipReady received a 403 for one bounded request using its GPTBot user-agent after the same fetched page returned a non-error response.
Why it matters
Different responses for a browser-like fetch and a supported AI crawler user-agent are deployment evidence to review. The comparison does not identify the responsible system or establish access behavior for every AI crawler or route.
How ShipReady detects it
What it inspects
For each successfully fetched page whose ordinary response is below 400, ShipReady makes one follow-redirects request to that final URL with its fixed GPTBot user-agent string.
What triggers a finding
The rule triggers only when the GPTBot user-agent probe receives HTTP 403. A non-403 answer does not trigger; network failures and rate-limit or unavailable responses are treated as inconclusive.
What the evidence shows
ShipReady observed a non-error ordinary fetch for the reported page and a 403 response to its one GPTBot user-agent probe of that final URL.
Limits of this check
ShipReady tests one fixed GPTBot user-agent, not every AI crawler, browser, route, authentication state, robots directive, CDN, WAF, or service policy. It does not establish why the 403 occurred, whether other crawlers are blocked, whether a crawler can access another route, or any citation, indexing, or legal outcome.
How to fix it
Confirm the intended edge policy
Review CDN, WAF, hosting, and application controls that can vary responses by user agent. Confirm whether the observed GPTBot response is intentional for this route.
Test the intended response behavior
After any configuration change, compare a normal request with the relevant crawler user-agent on representative routes. ShipReady's finding is based on one bounded GPTBot request.
How to verify the change
Compare the deployed responses
Request the reported URL as a normal client and using the intended crawler user-agent, following redirects in both cases.
Expected: The response behavior for the intended user-agent policy is deliberate and documented.