Skip to main content

Try Base free for 14 days, no card or M-Pesa needed See pricing

Guide

How generated URLs create crawl traps

Last updated
Page tools
Agent setup

Ndovu research on 14 Kenyan websites: which generated URL families and tool settings took crawlers’ time, which were real traps, and how to fix them.

A crawl trap is a part of a website that keeps producing new addresses, so crawlers keep requesting pages that add little. We scanned 14 Kenyan websites for the two shapes Ndovu’s Crawl Analytics looks for: URL families far larger than the site lists, and pages whose every request carries a query string.

Key findings

  1. 56% eachof verified crawler requests went to tool settings in query strings on two tool sites.
  2. 1 of 8large URL families were more than five times the size their sitemaps list: the rest were catalogues or need a closer look.
  3. 267,324addresses in four days in the largest family, a listed catalogue, not a trap.
  4. 33%of one store’s crawling went to paged shop and category listings with filter and sort parameters.

Fifth in Ndovu’s bot research series, after how bots crawl ecommerce websites and crawler bandwidth on error responses.

How we looked for traps

ShapeRule (four days, verified crawlers, pages only)
Large URL familyA first path segment (such as /items/) with 1,000+ distinct URLs requested and over 5% of the site’s crawling. A trap candidate if that is 5× or more the URLs the site’s sitemaps list under it.
Query-string pageOne path with 1,000+ requests, 80%+ of them with a query string, and over 5% of the site’s crawling.

These are the rules behind Crawl Analytics’ “endless set of URLs” issue, applied to four days instead of seven, so they are stricter here. Query values are never stored, so we count addresses by path; filter combinations on one path show as query-string requests.

Large URL families: most were catalogues

Large URL families and how much crawling they took

  • Trap candidate 1
  • Check against the catalogue 5
  • Listed catalogue 2
Reference pagesContent or tool site94% · ≤1.0× its listed size
Vehicle listingsOther website61% · 2.4× its listed size
Product comparison pagesStore38% · 4.0× its listed size
Product pagesStore37% · 2.3× its listed size
Generated item pagesContent or tool site34% · none in sitemaps
Product pagesStore33% · 3.2× its listed size
Item pagesContent or tool site19% · 0.8× its listed size
Product comparison pagesStore6% · 3.4× its listed size

Share of each site’s verified crawler requests (pages only) taken by the family, with addresses crawled ÷ addresses its sitemaps list. A trap candidate is 5× or more. Sites unnamed.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026; public sitemaps, 18 November 2026.

Copy linkDownload SVG
Embed
URL familySiteAddresses crawledListed in sitemapsCrawled ÷ listedNew on the last dayVerdict
Reference pagesContent or tool site267,324262,517+≤1.0×72%Listed catalogue
Vehicle listingsOther website7,3943,0712.4×20%Check against the catalogue
Product comparison pagesStore15,5063,8594.0×34%Check against the catalogue
Product pagesStore9,4844,1512.3×35%Check against the catalogue
Generated item pagesContent or tool site172,2920none listed67%Trap candidate
Product pagesStore1,0273243.2×13%Check against the catalogue
Item pagesContent or tool site1,9602,4180.8×87%Listed catalogue
Product comparison pagesStore2,4557223.4×30%Check against the catalogue

Table 1. URL families over the thresholds. “New on the last day” is the share of addresses crawled on 17 November not seen in the three days before. Sites are not named. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026; public sitemaps, 18 November 2026.

The biggest family, 267,324 addresses, matched pages the site lists, so it is a large catalogue that crawlers are working through. Size alone is not a trap.

The clear candidate was a family of generated item pages on a content or tool site: 172,292 addresses in four days, none in any sitemap, taking 34% of the site’s crawling. 67% of those crawled on the last day were new, so crawlers were still finding more. A page made for every possible value, each linking to more, looks exactly like this. New addresses alone prove nothing, though: the listed catalogues showed high rates too, because crawlers were still working through them.

The families in between, crawled at two to four times their listed size, need a closer look: old products and listings, variant and comparison URLs, or pages the sitemap leaves out.

Tool settings and filters in query strings

What the query strings encodeSitePagesRequestsShare of the site’s crawling
Tool settingsContent or tool site3325,35156.2%
Tool settingsContent or tool site47,58155.8%
Homepage query stringsStore11,52829.2%
Cart linksStore12,02326.2%
Shop listing filtersStore314,37519.9%
Category filtersStore29,37412.9%
Homepage query stringsStore11,4037.2%
Shop listing filtersStore16,1046.0%

Table 2. Paths where nearly every request carried a query string, grouped. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026; public sitemaps, 18 November 2026.

On two tool sites, a handful of tool pages took over half of all verified crawler requests (56% each), almost every one with different settings in the address. Each colour, size or option a tool writes into its link becomes a new page to a crawler. Sharing a configured result can be useful; offering every combination as a link is a trap.

On stores, paged shop and category listings carried filter and sort parameters on every request. Page 2 sorted by price, page 2 sorted by date and page 3 filtered by brand are different addresses with the same products. On one store, cart links with query strings took 26% of crawling: add-to-cart buttons are often plain links, so crawlers follow them.

How a small interface creates thousands of URLs

A category with 8 brands, 6 colours, 5 price bands, 4 sort orders and 10 pages allows 8 × 6 × 5 × 4 × 10 = 9,600 addresses, before multiple selections. That illustration is finite; these patterns are not:

PatternHow it growsSource
Filters in any order?brand=a&colour=b and ?colour=b&brand=a can both exist, and a filter can be appended twice.Google
Tool settingsEvery input value written into the address is a new URL.Our data above
CalendarsA next-month link that never ends.Google
Session and tracking IDsThe same page under a new parameter for every visit.Google
Broken relative linksA link to “catalog/” on /catalog/ leads to /catalog/catalog/, and on, if the server answers every path.Google
Pagination with no endPage 51 of a 50-page list still returns 200 and a “next” link.Google

A redirect loop is different again: the redirects return to an address already visited and never finish (MDN). It needs the chain of redirect targets to prove, which a request count cannot show.

Confirm a suspected trap

StepWhat to check
1. Compare with what you listDistinct URLs crawled in the family against the pages in your sitemaps, catalogue and Search Console. A gap in your inventory can look like a trap.
2. Watch for new addressesAre crawlers still finding URLs they have not requested before, day after day?
3. Find the link that makes themInspect the HTML and rendered links. Links added by JavaScript count if they are normal <a href> links (Google).
4. Look at what the URLs returnStatus, canonical, main content and outgoing links for a sample: real pages, duplicates, empty results or invalid values?

Crawl Analytics raises the issue for you and lists the family, sample URLs and crawlers; the filtered URL guide shows how to measure a parameter family.

Fix the rule that generates the URLs

  • Replace a parameter instead of appending it again, and keep one parameter order.
  • Validate values, end pagination at the last real page, and give calendars a range.
  • Return a 404 for empty or invalid combinations, at the URL itself (Google).
  • Stop linking states that should not be pages, such as every tool setting, while keeping links to the pages you want found.
ControlUse it forLimit
Change the templateStop generating or linking unwanted URLsKnown URLs may still be requested for a while
robots.txtAsk crawlers to skip a patternNot a way to keep pages out of Google, and crawlers choose to obey (Google)
noindexKeep a fetched page out of the indexOnly works if robots.txt lets the page be crawled (Google)
CanonicalPoint duplicates at the preferred URLThe duplicate is still fetched; the generator remains
404 or 410Addresses that should not existDoes not stop links creating new ones

Do not expect a robots.txt block to send Google to your other pages: Google only shifts that crawling elsewhere if it was already hitting your site’s capacity limit (Google).

Measure the result

TrackWhy
New distinct URLs per day in the familyShows whether generation stopped
Requests and data per day in the familyShows whether crawling shifted
Requests to the pages you want foundShows the fix did not cut off real content
A boundary check (last page, invalid value, relative link)Proves the mechanism is fixed, not hidden

Compare the same log source and URL family before and after, and keep the raw counts beside any share.

Method and limitations

How it works
Sample14 Kenyan websites with server access logs, monitored by NdovuSEOLabs; not a random sample.
Period14–17 November 2026: four complete days in Kenyan time.
ScopeVerified crawlers only, pages only (no images, scripts or styles), NdovuBot excluded. API routes are not counted as page families.
Listed pagesURLs in each site’s public sitemaps under the family, counted on 18 November 2026. One very large sitemap set was counted only partly, so its figure is a minimum.
Not measuredSearch Console and catalogue inventories, the links that generated each URL, redirect loops, indexing effects and results after a fix.

Original research by NdovuSEOLabs, from Crawl Analytics data. You may cite these figures with a link to this page.

Next steps

Was this helpful?