Skip to main content

Try Base free for 14 days, no card or M-Pesa needed See pricing

Guide

Redirect chains, crawl loops and repeated requests: what the logs show

Last updated
Page tools
Agent setup

Ndovu research on 14 Kenyan websites: which bots got redirected, what tracing the most-hit redirects found, and why 13 of 14 sites rate-limited crawlers.

The same address keeps appearing in a crawler report, answered with a redirect or an error, and it looks like a bot is stuck. We checked 14 Kenyan websites’ logs, then traced the redirects bots hit most to see what actually happens.

Key findings

  1. 3.2%of bot requests were redirects; 4.4% on the median website.
  2. 0 chainsand no loops in the 116 most-redirected paths we traced: each was one hop to a working page.
  3. 44%of those redirects only added or removed a trailing slash.
  4. 13 of 14websites rate-limited crawlers with 429s, all but 1 to verified bots, and none to Googlebot.

Sixth in Ndovu’s bot research series; crawl traps and error responses are covered separately.

What the logs recorded

ResponseBot requestsShareWhat it is
Redirect (3xx except 304)53,9493.2%Sent to another address
Not modified (304)3,6670.22%A cache check, not a redirect
Too many requests (429)31,9801.9%Rate-limited

Table 1. 1,669,357 bot requests across 14 websites, NdovuBot excluded. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Redirect share varied from 0.01% to 17% of a website’s bot requests. A request count shows activity, not a chain: a chain needs each redirect’s destination in order, and Ndovu’s logs, like most access logs, do not record it.

Which bots got redirected most

Which bots got redirected most

  • Share of the bot’s requests redirected n=14 websites
ShapBot64% · 27,980 redirects
KeenableBot49% · 1,471 redirects
Googlebot16% · 6,436 redirects
GoogleOther14% · 279 redirects
PetalBot12% · 1,932 redirects
DuckDuckBot7% · 90 redirects
Meta-ExternalAds4% · 3,360 redirects

Share of each bot’s requests answered with a redirect (3xx except 304). Bots with 1,000+ requests and 2%+ redirected.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Copy linkDownload SVG
Embed

ShapBot was redirected on 64% of its requests, 71% of them on one store, where most of what it asked for now redirects. Googlebot was redirected on 16%, so old URLs in links, sitemaps or Google’s list of known pages still matter.

We traced the redirects bots hit most

For each website we took the paths bots requested most that were nearly always redirected and never carried a query string, 116 paths on 10 websites, and followed each one hop by hop on 18 November 2026.

ResultPaths
One redirect, then a working page (200)114
Two or more redirects (a chain)0
A loop0
Ended in an error0
No longer redirects2

Table 2. Live traces of the most-redirected paths. Each path drew a median of 10 redirected bot requests over four days (up to 457), from a median of 2 bots. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

No chains and no loops. Of the 114 that still redirect, 50 only added or removed a trailing slash and 64 moved to a new address. So the cost here is not a crawler stuck in a cycle: it is one extra request, every time a bot follows an old or slightly wrong link. The fix is the link, not the redirect.

Chain, loop or repeat?

PatternWhat happensHow to prove it
RepeatA bot requests an old address again and gets the same single redirect.Requests over time to one URL; the redirect is fine, the link is not.
Chain/old → /middle → /current: each hop is another request. Search Console counts every one (Google).Follow each Location header to the end.
Loop/a → /b → /a. It never finishes (MDN); Google stops after 10 hops (Google).An address repeats in the trace.
TrapNew addresses keep appearing, often answering 200.See crawl traps.

A common real loop: Cloudflare’s Flexible mode talks to your server over HTTP while the server forces HTTPS, so each answer sends it round again (Cloudflare).

A 304 is not a redirect

Logs often group 304 with redirects because it is a 3xx code. It is a cache check: the bot already has the file, and the reply has no body (MDN). Our websites answered 3,667 of them, the largest number to Googlebot (680). Keep 304s out of any redirect count; they are a good sign.

Rate limiting was almost everywhere

13 of 14 websites answered some bot requests with 429 Too Many Requests, 31,980 in all: GPTBot 17,006, Meta-ExternalAgent 12,236, Meta-ExternalAds 2,039. None went to Googlebot. A 429 tells a client it sent too many requests in a given time; a Retry-After header can say when to try again (MDN).

On content sites the 429s fell almost entirely on content pages. On stores they fell on images, scripts and styles, comparison pages, cart pages and product pages.

  • Know where the limit lives. On shared hosting it may be the host’s rule, not yours. Ask which crawlers it applies to and at what rate.
  • Never limit the crawlers you need. Google reads 429 as a sign your server is overloaded and slows down (Google).
  • Limit on purpose, cheaply. A 429 is the right answer to an AI crawler you want to slow; it costs almost no data. Stop linking cart pages and you remove the reason to limit them.

Investigate a repeated path

StepWhat to do
1. Pick the requestFull URL, time, method and response. Is the question a repeat, a chain, a loop or a rejection?
2. Trace itGET the URL without following redirects; record each status and Location until it ends, repeats or reaches 10 hops.
3. Check the destinationA final 200 is a status, not a result: is it the right product, listing or article?
4. Check every layerCDN, server, WordPress and plugins can each redirect; conflicting rules make loops.
5. Find the old linkMenus, internal links, sitemaps and feeds that still use the old or slash-less address.

Ndovu’s Crawl Analytics shows redirect counts by page and bot; its Activity view lists individual requests, errors first.

Method and limitations

How it works
Sample14 Kenyan websites with server access logs, monitored by NdovuSEOLabs; not a random sample.
Period14–17 November 2026 (logs); traces on 18 November 2026.
Traced pathsUp to 25 per website: bot requests 80%+ redirected, at least 5 redirects, never a query string. GET, redirects followed by hand, 10-hop limit.
429 detailTotals from the logs; where they fell from Ndovu’s retained Activity sample, which kept nearly all of them.
Not measuredRedirects with query strings, chains at the time of each request, Retry-After headers, and why each bot repeated a request.

Original research by NdovuSEOLabs, from Crawl Analytics data and live traces. You may cite these figures with a link to this page.

Next steps

Was this helpful?