Ndovu research on 20 Kenyan websites: how a CDN changes crawling and what your logs can see, and why a new log source can look like a bot spike.
You move a website behind a CDN and the crawler graph jumps. Google explained one reason on 11 November 2026: faster, steadier responses let Google crawl more, and a cold cache sends those requests straight to your server (Google Search Central). There is a second reason a graph can jump, which has nothing to do with crawlers: what your monitoring can see.
Key findings
- 13 of 20Kenyan websites we monitor sit behind Cloudflare; on 10 the homepage came from Cloudflare’s cache.
- 0 of 20websites’ log sources could see requests answered from the CDN cache: they never reach the server or WordPress.
- 1.4×the bot requests a server log recorded against the WordPress plugin on the same days (up to 3.1×): a new source looks like a spike.
- 1website’s Cloudflare challenged our checker, as it should; the question is whether it lets the crawlers you want through.
Seventh in Ndovu’s bot research series; see why CDN, server and WordPress counts differ for how each source works.
Why crawling can rise after a CDN
| Mechanism | What happens | Source |
|---|---|---|
| Faster responses | Google reads fast, stable responses as spare capacity and crawls more, gradually and only if it has pages it wants. | |
| Cold cache | A new CDN starts empty. Every uncached URL is a miss, so early crawling still lands on your server. | |
| HTML not cached | Cloudflare caches images, CSS and scripts by default, not HTML or JSON. Without a page-cache rule, every page request still reaches the origin. | Cloudflare |
| Every URL is its own entry | The default cache key includes the query string, so each filter, sort and tool setting is cached separately. | Cloudflare |
The last row is why crawl traps get worse behind a CDN: warming a category page does nothing for its thousands of filtered versions.
How our websites are set up
How the monitored websites serve their homepage
- Answered at the CDN edge
- Reaches the server
- Challenged
- No CDN
Each homepage requested twice on 18 November 2026; setup read from the response headers (Cloudflare’s HIT and DYNAMIC cache statuses, a challenge, or a server page-cache header).
Source: NdovuSEOLabs live check of 20 monitored Kenyan websites, 18 November 2026.
Half of these websites (10 of 20) cache pages at Cloudflare’s edge, so a bot fetching a cached page is answered there. That is good for the server, and invisible to the logs we studied: those count only requests that reached the server or WordPress. Two websites had set up Cloudflare log delivery that had not delivered anything yet, so the edge view was simply missing.
A new log source looks like a traffic spike
Twelve of our 14 server-log websites switched from the WordPress plugin to their server’s access log on 12 November, and the plugin kept running. On 13 websites both sources recorded every day, so we can compare them directly:
| Website | WordPress plugin | Server log | Server log ÷ plugin | Errors: plugin → server log |
|---|---|---|---|---|
| 1. Store | 4,652 | 14,372 | 3.09× | 76 → 227 |
| 2. Other website | 93,912 | 227,009 | 2.42× | 2,075 → 3,622 |
| 3. Store | 578 | 1,175 | 2.03× | 365 → 380 |
| 4. Store | 24,955 | 44,272 | 1.77× | 5,280 → 6,046 |
| 5. Store | 5,750 | 8,540 | 1.49× | 11 → 94 |
| 6. Content or tool site | 15,093 | 21,597 | 1.43× | 20 → 563 |
| 7. Store | 692 | 941 | 1.36× | 441 → 442 |
| 8. Store | 129,163 | 167,207 | 1.29× | 2,587 → 9,860 |
| 9. Store | 69,543 | 83,727 | 1.20× | 4 → 2,073 |
| 10. Other website | 4,314 | 5,120 | 1.19× | 178 → 234 |
| 11. Store | 11,078 | 12,549 | 1.13× | 3,306 → 3,702 |
| 12. Content or tool site | 489,005 | 498,634 | 1.02× | 3,981 → 17,852 |
| 13. Content or tool site | 571,572 | 582,653 | 1.02× | 205 → 12,487 |
Table 1. Bot requests recorded by each source for the same websites on the same days, NdovuBot excluded. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.
Same websites, same days: server log ÷ WordPress plugin
- Server log ÷ plugin n=13 websites
- Median 1.4×
Bot requests recorded by the server-log source divided by those the WordPress plugin recorded, for websites where both ran every day. 1× would mean the same count. NdovuBot excluded.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.
Every website looked busier in its server log: a median 1.4× and up to 3.1×. The server log adds images, files, cached pages the server answers itself, and errors WordPress never sees: on one website errors went from 4 to 2,073. A graph that changes source mid-way shows a step that no crawler made. Compare like with like, or start the comparison after the switch.
Bot protection can block the crawlers you want
Google lists two ways a CDN can stop it: hard blocks (503, 429, timeouts) and soft blocks, such as an “are you human” challenge, or an error page served with a 200 (Google). A 200 is not proof the crawler got your page.
- Verify, then allow. Let verified crawlers through using the operator’s published IP ranges, not the user-agent name (Google).
- Know which feature acted. Cloudflare’s Bot Fight Mode cannot be skipped with custom rules, unlike its WAF rules (Cloudflare).
- Check from the crawler’s side. A page that opens in your browser can still challenge a bot; inspect the response the crawler got, and URL Inspection’s rendered page.
On the website that challenged our checker, verified crawlers kept reaching the server in large numbers, so its protection was letting them through. That is the outcome to confirm after any rule change.
Diagnose a spike across four layers
| Layer | Look at | It answers |
|---|---|---|
| CDN edge | Requests by verified crawler, status, cache status, security action | What arrived, and what was answered or blocked before the server |
| Server | Requests, response times, errors, CPU and memory | How much work reached the server, and could it cope |
| Application | URL families, page generation, database time, returned content | Which requests were expensive or wrong |
| Measurement | Source start dates, coverage, sampling, rule changes | Whether before and after are comparable |
- Edge requests up, server flat: the cache is absorbing it.
- Server requests and response times up together: check what is not cached and which URL families are new.
- Server down, edge challenges up: wanted crawlers may be blocked.
- A jump on the day a source changed: measure coverage before blaming crawlers.
Before you switch a CDN on
| Step | Why |
|---|---|
| Decide which pages can be cached | Products, categories, listings and articles, yes; carts, checkout and accounts, no. Test real headers, not plugin settings. |
| Warm the important URLs slowly | Google suggests pre-fetching key pages at a controlled pace before switching traffic (Google). Warming is traffic too. |
| Shrink the URL space | Each filter and tool setting is its own cache entry; stop linking the ones that should not be pages. |
| Test crawler access | Key pages, robots.txt and assets, as verified crawlers, after every rule change. |
| Keep both views | Edge logs and server logs, with the cutover time written down. |
If the server still struggles
Return 429 or 503 for a few hours, no more than one to two days, optionally with Retry-After (Google). It slows Google across the whole hostname, and URLs that keep failing for several days may drop out of the index. Do not use 401 or 403 for this (Google). Our redirects and rate-limiting study found 13 of 14 websites already sending 429s.
Method and limitations
| How it works | |
|---|---|
| Two-source comparison | 13 Kenyan websites that ran both a server-log source and the WordPress plugin on every day of 14–17 November 2026, NdovuBot excluded. |
| CDN check | Each of 20 websites’ homepage requested twice on 18 November 2026 with a plain research user-agent; CDN and cache status from response headers. One page per website. |
| Coverage | Source start dates from Ndovu’s records. No source recorded a CDN cache status. |
| Not measured | CDN cutover dates, before-and-after crawling, edge request counts and server load. We did not observe a CDN migration in this period. |
Original research by NdovuSEOLabs, from Crawl Analytics data and live checks. You may cite these figures with a link to this page.