Advertisement
Content Moved? Use Search to Locate
Detailed view of network cables plugged into a server rack in a data center.

How to Check Googlebot Access After a CDN Migration

After a CDN cutover, check the same important URLs before and after the change, inspect what the CDN and WAF did with requests, and compare cache-miss activity with origin health. Verify any requests attributed to Google using Google’s documented methods—not a Googlebot user-agent string alone—and use Search Console as one part of the diagnosis. This audit is for the site owner or developer responsible for the migration; a successful test of a few URLs does not establish sitewide access or prove that pages are indexed.

Set a baseline before the cutover

Choose a manageable sample that reflects how people and search crawlers reach the site: the homepage, several representative service or product pages, and important landing pages. This is a practical sampling suggestion, not a Google requirement. The right sample and the time you spend monitoring depend on the site, its cache configuration, traffic, and the capacity of its origin server.

For each URL, record the response status and whether the returned content is the intended page. Note redirects, unexpected errors, and any challenge or interstitial page. If available, record the relevant CDN or WAF action, cache status, origin request, and origin health at the time of the check. Keep timestamps and time zones consistent across systems so you can compare events later. A test through the current site path gives you a baseline; it does not by itself verify how a crawler will be treated.

Repeat the checks through the new CDN

After cutover, request the same URLs and compare the status, content, and behavior with your baseline. Confirm that each page returns its intended content through the new edge—not just that the homepage loads. Look for challenge pages, unexpected redirects, blocked resources relevant to the page, and errors that appear only on particular paths. Check the CDN and WAF logs for the action taken on each request and whether the response was served from cache or required an origin fetch.

Google Search Central’s Moving to a CDN without overwhelming your origin (or blocking Googlebot), published November 11, 2026, describes operational risks around CDN migrations; it does not announce a new Googlebot rule or require a particular migration procedure. Google notes that a cold-cache request may need to fetch a URL’s response from the origin, while later cache hits may be served at the edge. Do not assume that the CDN has immediately reduced origin load, or that a particular cache-warming schedule is required. If you choose to pre-warm selected URLs, make the pace appropriate to your cache setup and origin capacity.

Verify requests attributed to Google

Review CDN/WAF and server logs for requests labeled as Googlebot, especially requests associated with a challenge, block, or error. A user-agent string can be spoofed, so it is not proof that the request came from Google. Google’s Verify requests from Google crawlers and fetchers documents two verification approaches: perform a reverse-DNS lookup on the request’s IP address and then a forward-DNS lookup on the resulting hostname, or compare the IP against Google’s published ranges. Google’s List of Google’s common crawlers provides common-crawler IP ranges and reverse-DNS information.

Keep the crawler type in view when interpreting a log entry. Google distinguishes common crawlers used for products such as Search from user-triggered fetchers; not every Google-associated request represents Search crawling, and Google notes that user-triggered fetchers may ignore robots.txt. Verify the request type and identity using the applicable Google documentation before changing an edge rule.

Compare edge activity with origin health

For the same time period, line up CDN/WAF events, cache misses, requests sent upstream, origin response statuses, and origin availability or health signals. Check whether a verified crawler request was challenged at the edge, whether a cache miss led to an origin request, and whether origin errors or availability problems occurred at the same time. A timestamp match can help narrow the investigation, but it does not by itself prove which system caused the problem; review the request path and relevant logs together.

If verified Google crawler requests receive an inappropriate challenge or error, investigate the specific edge rule and adjust handling for verified crawlers where appropriate. Google warns that blanket bot challenges can obstruct crawlers and discusses allowing verified crawlers, targeting unwanted bots, and pre-warming selected URLs as possible considerations. If cache misses coincide with origin health problems, investigate origin capacity and cache behavior rather than assuming that the CDN is reducing load. Google’s discussion of temporary 429 or 503 responses is emergency guidance for critical origin load over a short window—not a routine cutover setting.

Use Search Console for corroboration, not blanket proof

Google’s Troubleshoot Google Search crawling errors guide explains that Crawl Stats can show crawl history and host-availability issues. Review the report around the cutover for availability patterns, then compare relevant timestamps with your CDN, WAF, and origin logs. URL Inspection can test selected URLs and may report an issue such as “Hostload exceeded,” but a successful result applies to the URL you checked. It does not prove that every page, resource, crawler type, or future request will work.

Search Console does not provide crawl history filtered by URL or path. For evidence about whether a particular page was requested, inspect server logs and match the path and timestamp; verify any request attributed to Google before treating it as a Google crawler. Crawling is not the same as indexing, so a page being fetched does not establish that it is indexed. Keep access diagnosis separate from index-status checks.

Hypothetical example: the product page gets a challenge

Suppose the homepage loads after cutover, but a product URL appears to return a challenge page at the CDN. First, request that exact URL and confirm the status and returned content. Then inspect the CDN/WAF action and rule for that path, and check whether the event was a cache hit or miss and whether it generated an origin request. If logs attribute a request to Google, verify its IP with reverse DNS followed by forward DNS, or against Google’s published IP ranges. You can also test the selected URL with URL Inspection and compare the relevant timestamps with origin health and server logs. These checks help distinguish an edge challenge from an origin problem; the homepage result alone cannot settle the product-page issue.

CDN cutover verification checklist

  • Before and after cutover, check the same representative homepage, service or product, and high-value landing-page URLs.
  • For each URL, confirm the intended status and content; note unexpected redirects, errors, or challenge pages.
  • Review CDN/WAF actions, cache status, upstream requests, and origin health using comparable timestamps.
  • Verify suspicious Google-attributed requests with reverse DNS followed by forward DNS, or Google’s published IP ranges; do not rely on user-agent text alone.
  • Use Crawl Stats for crawl and host-availability patterns, URL Inspection for selected URLs, and server logs for path-level evidence.
  • Decide whether the evidence points to an edge rule to investigate, origin capacity or cache behavior to examine, or further monitoring. Keep crawl access separate from index status.

If you want help turning this audit into an ongoing operating process, Splinternet’s managed website hosting service can support CDN and WAF implementation, origin monitoring, and continued technical checks. The aim is to make configuration changes measurable and maintainable, not to treat one successful test as proof that every path is healthy.

Sources

Editorial note: AI assists with research, drafting and automated checks. Sources are linked so you can verify the guidance. Platform requirements can change; confirm the details that apply to your setup.