Ndovu research on 14 Kenyan websites: which generated URL families and tool settings took crawlers’ time, which were real traps, and how to fix them.
A crawl trap is a part of a website that keeps producing new addresses, so crawlers keep requesting pages that add little. We scanned 14 Kenyan websites for the two shapes Ndovu’s Crawl Analytics looks for: URL families far larger than the site lists, and pages whose every request carries a query string.
Key findings
- 56% eachof verified crawler requests went to tool settings in query strings on two tool sites.
- 1 of 8large URL families were more than five times the size their sitemaps list: the rest were catalogues or need a closer look.
- 267,324addresses in four days in the largest family, a listed catalogue, not a trap.
- 33%of one store’s crawling went to paged shop and category listings with filter and sort parameters.
Fifth in Ndovu’s bot research series, after how bots crawl ecommerce websites and crawler bandwidth on error responses.
How we looked for traps
| Shape | Rule (four days, verified crawlers, pages only) |
|---|---|
| Large URL family | A first path segment (such as /items/) with 1,000+ distinct URLs requested and over 5% of the site’s crawling. A trap candidate if that is 5× or more the URLs the site’s sitemaps list under it. |
| Query-string page | One path with 1,000+ requests, 80%+ of them with a query string, and over 5% of the site’s crawling. |
These are the rules behind Crawl Analytics’ “endless set of URLs” issue, applied to four days instead of seven, so they are stricter here. Query values are never stored, so we count addresses by path; filter combinations on one path show as query-string requests.
Large URL families: most were catalogues
Large URL families and how much crawling they took
- Trap candidate 1
- Check against the catalogue 5
- Listed catalogue 2
Share of each site’s verified crawler requests (pages only) taken by the family, with addresses crawled ÷ addresses its sitemaps list. A trap candidate is 5× or more. Sites unnamed.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026; public sitemaps, 18 November 2026.
| URL family | Site | Addresses crawled | Listed in sitemaps | Crawled ÷ listed | New on the last day | Verdict |
|---|---|---|---|---|---|---|
| Reference pages | Content or tool site | 267,324 | 262,517+ | ≤1.0× | 72% | Listed catalogue |
| Vehicle listings | Other website | 7,394 | 3,071 | 2.4× | 20% | Check against the catalogue |
| Product comparison pages | Store | 15,506 | 3,859 | 4.0× | 34% | Check against the catalogue |
| Product pages | Store | 9,484 | 4,151 | 2.3× | 35% | Check against the catalogue |
| Generated item pages | Content or tool site | 172,292 | 0 | none listed | 67% | Trap candidate |
| Product pages | Store | 1,027 | 324 | 3.2× | 13% | Check against the catalogue |
| Item pages | Content or tool site | 1,960 | 2,418 | 0.8× | 87% | Listed catalogue |
| Product comparison pages | Store | 2,455 | 722 | 3.4× | 30% | Check against the catalogue |
Table 1. URL families over the thresholds. “New on the last day” is the share of addresses crawled on 17 November not seen in the three days before. Sites are not named. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026; public sitemaps, 18 November 2026.
The biggest family, 267,324 addresses, matched pages the site lists, so it is a large catalogue that crawlers are working through. Size alone is not a trap.
The clear candidate was a family of generated item pages on a content or tool site: 172,292 addresses in four days, none in any sitemap, taking 34% of the site’s crawling. 67% of those crawled on the last day were new, so crawlers were still finding more. A page made for every possible value, each linking to more, looks exactly like this. New addresses alone prove nothing, though: the listed catalogues showed high rates too, because crawlers were still working through them.
The families in between, crawled at two to four times their listed size, need a closer look: old products and listings, variant and comparison URLs, or pages the sitemap leaves out.
Tool settings and filters in query strings
| What the query strings encode | Site | Pages | Requests | Share of the site’s crawling |
|---|---|---|---|---|
| Tool settings | Content or tool site | 3 | 325,351 | 56.2% |
| Tool settings | Content or tool site | 4 | 7,581 | 55.8% |
| Homepage query strings | Store | 1 | 1,528 | 29.2% |
| Cart links | Store | 1 | 2,023 | 26.2% |
| Shop listing filters | Store | 3 | 14,375 | 19.9% |
| Category filters | Store | 2 | 9,374 | 12.9% |
| Homepage query strings | Store | 1 | 1,403 | 7.2% |
| Shop listing filters | Store | 1 | 6,104 | 6.0% |
Table 2. Paths where nearly every request carried a query string, grouped. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026; public sitemaps, 18 November 2026.
On two tool sites, a handful of tool pages took over half of all verified crawler requests (56% each), almost every one with different settings in the address. Each colour, size or option a tool writes into its link becomes a new page to a crawler. Sharing a configured result can be useful; offering every combination as a link is a trap.
On stores, paged shop and category listings carried filter and sort parameters on every request. Page 2 sorted by price, page 2 sorted by date and page 3 filtered by brand are different addresses with the same products. On one store, cart links with query strings took 26% of crawling: add-to-cart buttons are often plain links, so crawlers follow them.
How a small interface creates thousands of URLs
A category with 8 brands, 6 colours, 5 price bands, 4 sort orders and 10 pages allows 8 × 6 × 5 × 4 × 10 = 9,600 addresses, before multiple selections. That illustration is finite; these patterns are not:
| Pattern | How it grows | Source |
|---|---|---|
| Filters in any order | ?brand=a&colour=b and ?colour=b&brand=a can both exist, and a filter can be appended twice. | |
| Tool settings | Every input value written into the address is a new URL. | Our data above |
| Calendars | A next-month link that never ends. | |
| Session and tracking IDs | The same page under a new parameter for every visit. | |
| Broken relative links | A link to “catalog/” on /catalog/ leads to /catalog/catalog/, and on, if the server answers every path. | |
| Pagination with no end | Page 51 of a 50-page list still returns 200 and a “next” link. |
A redirect loop is different again: the redirects return to an address already visited and never finish (MDN). It needs the chain of redirect targets to prove, which a request count cannot show.
Confirm a suspected trap
| Step | What to check |
|---|---|
| 1. Compare with what you list | Distinct URLs crawled in the family against the pages in your sitemaps, catalogue and Search Console. A gap in your inventory can look like a trap. |
| 2. Watch for new addresses | Are crawlers still finding URLs they have not requested before, day after day? |
| 3. Find the link that makes them | Inspect the HTML and rendered links. Links added by JavaScript count if they are normal <a href> links (Google). |
| 4. Look at what the URLs return | Status, canonical, main content and outgoing links for a sample: real pages, duplicates, empty results or invalid values? |
Crawl Analytics raises the issue for you and lists the family, sample URLs and crawlers; the filtered URL guide shows how to measure a parameter family.
Fix the rule that generates the URLs
- Replace a parameter instead of appending it again, and keep one parameter order.
- Validate values, end pagination at the last real page, and give calendars a range.
- Return a 404 for empty or invalid combinations, at the URL itself (Google).
- Stop linking states that should not be pages, such as every tool setting, while keeping links to the pages you want found.
| Control | Use it for | Limit |
|---|---|---|
| Change the template | Stop generating or linking unwanted URLs | Known URLs may still be requested for a while |
| robots.txt | Ask crawlers to skip a pattern | Not a way to keep pages out of Google, and crawlers choose to obey (Google) |
| noindex | Keep a fetched page out of the index | Only works if robots.txt lets the page be crawled (Google) |
| Canonical | Point duplicates at the preferred URL | The duplicate is still fetched; the generator remains |
| 404 or 410 | Addresses that should not exist | Does not stop links creating new ones |
Do not expect a robots.txt block to send Google to your other pages: Google only shifts that crawling elsewhere if it was already hitting your site’s capacity limit (Google).
Measure the result
| Track | Why |
|---|---|
| New distinct URLs per day in the family | Shows whether generation stopped |
| Requests and data per day in the family | Shows whether crawling shifted |
| Requests to the pages you want found | Shows the fix did not cut off real content |
| A boundary check (last page, invalid value, relative link) | Proves the mechanism is fixed, not hidden |
Compare the same log source and URL family before and after, and keep the raw counts beside any share.
Method and limitations
| How it works | |
|---|---|
| Sample | 14 Kenyan websites with server access logs, monitored by NdovuSEOLabs; not a random sample. |
| Period | 14–17 November 2026: four complete days in Kenyan time. |
| Scope | Verified crawlers only, pages only (no images, scripts or styles), NdovuBot excluded. API routes are not counted as page families. |
| Listed pages | URLs in each site’s public sitemaps under the family, counted on 18 November 2026. One very large sitemap set was counted only partly, so its figure is a minimum. |
| Not measured | Search Console and catalogue inventories, the links that generated each URL, redirect loops, indexing effects and results after a fix. |
Original research by NdovuSEOLabs, from Crawl Analytics data. You may cite these figures with a link to this page.