When the Edge Says No: Diagnosing Crawl Blocks on UK Sites Behind CDNs and WAFs
CDNs and web application firewalls protect UK sites, but they can also challenge, throttle, or block verified crawlers before a request reaches the origin. This audit framework uses Search Console crawl stats, edge analytics, origin logs, and staged fetch tests to find crawl-access failures and prioritise fixes by revenue and indexation risk.