Google published an explainer that discusses how Content Delivery Networks (CDNs) influence search crawling and improve SEO but also how they can sometimes cause problems.
What Is a CDN?
A Content Delivery Network (CDN) caches web pages and serves them from the nearest data center, reducing load time by minimizing the distance between the server and the visitor’s browser. This enhances speed and performance.
CDNs and Enhanced Crawling
Using a CDN (Content Delivery Network) enhances Google’s crawl rate by raising the throttling threshold, which limits crawling when servers slow down. When Google detects web pages served via a CDN, it increases crawling, allowing more pages to be indexed. This makes CDNs an attractive choice for SEOs and publishers aiming to optimize crawling efficiency and page indexing.
When using a CDN, the first time pages are served, they must be delivered from your server, as illustrated by Google’s example of a site with over a million pages.
When a URL is accessed for the first time, the CDN’s cache is “cold,” meaning the URL’s content hasn’t been cached yet. Consequently, the origin server must serve the URL at least once to “warm up” the CDN’s cache, similar to how HTTP caching works. For instance, if a webshop has 1,000,007 URLs, the server must serve each URL initially before the CDN’s cache becomes effective. This process can significantly impact your crawl budget, resulting in a higher crawl rate for a few days. Keep this in mind when launching a large number of URLs simultaneously.
When CDNs Backfire for Crawling
CDNs can sometimes block Googlebot, hindering crawling and indexing. This issue occurs through two types of blocks:
1. Hard Blocks
- Server Errors: Responses like 500 (internal server error) or 502 (bad gateway) slow Googlebot’s crawl rate. Prolonged errors may lead to URLs being dropped from Google’s search index.
- Preferred Response: A 503 (service unavailable) indicates a temporary issue and avoids long-term damage.
- Random Errors: Incorrect 200 response codes (indicating a successful response but delivering error pages) are problematic. Google treats these as duplicates, dropping them from the index, with recovery taking significant time.
2. Soft Blocks
- Bot Interstitials: Pop-ups like “Are you human?” can block Googlebot. These should send a 503 server response to signal a temporary issue and prevent indexing problems.
Proper CDN configuration is critical to avoid these crawling issues and ensure optimal website performance.
Google’s new documentation explains:
“…when the interstitial shows up, that’s all they see, not your awesome site. In case of these bot-verification interstitials, we strongly recommend sending a clear signal in the form of a 503 HTTP status code to automated clients like crawlers that the content is temporarily unavailable. This will ensure that the content is not removed from Google’s index automatically.”
Troubleshooting with the URL Inspection Tool and WAF Controls
Google suggests using the URL Inspection Tool in Search Console to verify how your CDN serves web pages. If your Web Application Firewall (WAF) blocks Googlebot by IP address, you can cross-check the blocked IPs against Google’s official list of IP addresses to identify and resolve issues.
This approach helps ensure Googlebot has uninterrupted access to crawl your web pages, maintaining optimal site performance and search engine visibility which help to SEO.
Google offers the following CDN-level debugging advice:
“If you need your site to show up in search engines, we strongly recommend checking whether the crawlers you care about can access your site. Remember that the IPs may end up on a blocklist automatically, without you knowing, so checking in on the blocklists every now and then is a good idea for your site’s success in search and beyond. If the blocklist is very long (not unlike this blog post), try to look for just the first few segments of the IP ranges, for example, instead of looking for 192.168.0.101 you can just look for 192.168.”
Read Google’s documentation for more information:
Crawling December: CDNs and crawling
