Ndovu research on 14 Kenyan websites: which bots got redirected, what tracing the most-hit redirects found, and why 13 of 14 sites rate-limited crawlers.
The same address keeps appearing in a crawler report, answered with a redirect or an error, and it looks like a bot is stuck. We checked 14 Kenyan websites’ logs, then traced the redirects bots hit most to see what actually happens.
Key findings
- 3.2%of bot requests were redirects; 4.4% on the median website.
- 0 chainsand no loops in the 116 most-redirected paths we traced: each was one hop to a working page.
- 44%of those redirects only added or removed a trailing slash.
- 13 of 14websites rate-limited crawlers with 429s, all but 1 to verified bots, and none to Googlebot.
Sixth in Ndovu’s bot research series; crawl traps and error responses are covered separately.
What the logs recorded
| Response | Bot requests | Share | What it is |
|---|---|---|---|
| Redirect (3xx except 304) | 53,949 | 3.2% | Sent to another address |
| Not modified (304) | 3,667 | 0.22% | A cache check, not a redirect |
| Too many requests (429) | 31,980 | 1.9% | Rate-limited |
Table 1. 1,669,357 bot requests across 14 websites, NdovuBot excluded. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.
Redirect share varied from 0.01% to 17% of a website’s bot requests. A request count shows activity, not a chain: a chain needs each redirect’s destination in order, and Ndovu’s logs, like most access logs, do not record it.
Which bots got redirected most
Which bots got redirected most
- Share of the bot’s requests redirected n=14 websites
Share of each bot’s requests answered with a redirect (3xx except 304). Bots with 1,000+ requests and 2%+ redirected.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.
ShapBot was redirected on 64% of its requests, 71% of them on one store, where most of what it asked for now redirects. Googlebot was redirected on 16%, so old URLs in links, sitemaps or Google’s list of known pages still matter.
We traced the redirects bots hit most
For each website we took the paths bots requested most that were nearly always redirected and never carried a query string, 116 paths on 10 websites, and followed each one hop by hop on 18 November 2026.
| Result | Paths |
|---|---|
| One redirect, then a working page (200) | 114 |
| Two or more redirects (a chain) | 0 |
| A loop | 0 |
| Ended in an error | 0 |
| No longer redirects | 2 |
Table 2. Live traces of the most-redirected paths. Each path drew a median of 10 redirected bot requests over four days (up to 457), from a median of 2 bots. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.
No chains and no loops. Of the 114 that still redirect, 50 only added or removed a trailing slash and 64 moved to a new address. So the cost here is not a crawler stuck in a cycle: it is one extra request, every time a bot follows an old or slightly wrong link. The fix is the link, not the redirect.
Chain, loop or repeat?
| Pattern | What happens | How to prove it |
|---|---|---|
| Repeat | A bot requests an old address again and gets the same single redirect. | Requests over time to one URL; the redirect is fine, the link is not. |
| Chain | /old → /middle → /current: each hop is another request. Search Console counts every one (Google). | Follow each Location header to the end. |
| Loop | /a → /b → /a. It never finishes (MDN); Google stops after 10 hops (Google). | An address repeats in the trace. |
| Trap | New addresses keep appearing, often answering 200. | See crawl traps. |
A common real loop: Cloudflare’s Flexible mode talks to your server over HTTP while the server forces HTTPS, so each answer sends it round again (Cloudflare).
A 304 is not a redirect
Logs often group 304 with redirects because it is a 3xx code. It is a cache check: the bot already has the file, and the reply has no body (MDN). Our websites answered 3,667 of them, the largest number to Googlebot (680). Keep 304s out of any redirect count; they are a good sign.
Rate limiting was almost everywhere
13 of 14 websites answered some bot requests with 429 Too Many Requests, 31,980 in all: GPTBot 17,006, Meta-ExternalAgent 12,236, Meta-ExternalAds 2,039. None went to Googlebot. A 429 tells a client it sent too many requests in a given time; a Retry-After header can say when to try again (MDN).
On content sites the 429s fell almost entirely on content pages. On stores they fell on images, scripts and styles, comparison pages, cart pages and product pages.
- Know where the limit lives. On shared hosting it may be the host’s rule, not yours. Ask which crawlers it applies to and at what rate.
- Never limit the crawlers you need. Google reads 429 as a sign your server is overloaded and slows down (Google).
- Limit on purpose, cheaply. A 429 is the right answer to an AI crawler you want to slow; it costs almost no data. Stop linking cart pages and you remove the reason to limit them.
Investigate a repeated path
| Step | What to do |
|---|---|
| 1. Pick the request | Full URL, time, method and response. Is the question a repeat, a chain, a loop or a rejection? |
| 2. Trace it | GET the URL without following redirects; record each status and Location until it ends, repeats or reaches 10 hops. |
| 3. Check the destination | A final 200 is a status, not a result: is it the right product, listing or article? |
| 4. Check every layer | CDN, server, WordPress and plugins can each redirect; conflicting rules make loops. |
| 5. Find the old link | Menus, internal links, sitemaps and feeds that still use the old or slash-less address. |
Ndovu’s Crawl Analytics shows redirect counts by page and bot; its Activity view lists individual requests, errors first.
Method and limitations
| How it works | |
|---|---|
| Sample | 14 Kenyan websites with server access logs, monitored by NdovuSEOLabs; not a random sample. |
| Period | 14–17 November 2026 (logs); traces on 18 November 2026. |
| Traced paths | Up to 25 per website: bot requests 80%+ redirected, at least 5 redirects, never a query string. GET, redirects followed by hand, 10-hop limit. |
| 429 detail | Totals from the logs; where they fell from Ndovu’s retained Activity sample, which kept nearly all of them. |
| Not measured | Redirects with query strings, chains at the time of each request, Retry-After headers, and why each bot repeated a request. |
Original research by NdovuSEOLabs, from Crawl Analytics data and live traces. You may cite these figures with a link to this page.