Skip to main content

Try Base free for 14 days, no card or M-Pesa needed See pricing

Guide

How to investigate crawler activity on filtered and faceted URLs

Last updated
Page tools
Agent setup

Filters for make, price, location and sort can multiply your URLs. Measure what crawlers request, separate useful filtered pages from the rest, then act.

Your car catalogue lets buyers filter by make, model, price and location, and sort the results. Your logs now show Googlebot asking for URLs like /cars/?sort=price_asc&location=nakuru&price=1m-2m.

Some filtered pages are worth finding in search. Most combinations are not. Measure before you block anything.

The short version. Group crawler requests by parameter pattern. For each pattern, ask whether a searcher would want that page, then compare that with how much crawling it takes. Give useful filters a clean, linked URL; keep the rest out of the crawl; return 404 for empty combinations.

How a few filters become thousands of URLs

Each optional filter multiplies the others. With 30 makes, 8 price bands, 12 locations and 4 sort orders, /cars/ alone can produce (30+1) × (8+1) × (12+1) × (4+1) = 18,135 URLs. Add models, page numbers and parameters in a different order, and the count keeps growing.

Google names two costs: crawlers request a very large number of these URLs, and the time spent on them leaves less for new, useful pages (Google).

What Google recommends

If the filtered URLs…Google’s options
Don’t need to be in searchDisallow them in robots.txt; or put filters after a # fragment, which Google generally ignores; or point them to the unfiltered page with rel="canonical" (may reduce their crawling over time, less effective long term); or add rel="nofollow" to every link to them.
Should be crawled and indexedUse & between parameters, keep filters in a consistent order, prevent duplicate filters, and return a 404 for combinations with no results, duplicate filters, nonsense values or pages past the last one.

Source for both rows: Google’s faceted navigation guide. Two side effects to plan for: robots.txt manages crawling and does not keep a URL out of Google, and a blocked URL linked from elsewhere can still appear without a description (Google). And a noindex only works if the page is not blocked in robots.txt (Google), so don’t combine the two on one pattern.

Step 1 · Collect the requests

Use verified search crawler requests from your server or CDN logs, at least a week of them. A small sample looks like this:

time              crawler    verified  path                                            status
2026-11-10 02:14  Googlebot  yes       /cars/toyota/                                   200
2026-11-10 02:15  Googlebot  yes       /cars/?make=toyota                              200
2026-11-10 02:15  Googlebot  yes       /cars/?make=toyota&sort=price_asc               200
2026-11-10 02:16  Googlebot  yes       /cars/?sort=price_asc&make=toyota               200
2026-11-10 03:40  Googlebot  yes       /cars/?location=mombasa                         200
2026-11-10 03:41  Googlebot  yes       /cars/?location=mombasa&price=500k-1m&sort=new  200
2026-11-10 05:02  Googlebot  yes       /cars/?make=subaru&model=demio                  200
2026-11-10 05:03  Googlebot  yes       /cars/?make=toyota&make=toyota                  200
2026-11-10 06:20  Googlebot  yes       /cars/?make=toyota&page=48                      200
2026-11-11 02:14  Googlebot  yes       /cars/?make=toyota                              200
2026-11-11 02:16  Googlebot  yes       /cars/?make=toyota&sort=price_asc               200
2026-11-11 09:12  Bingbot    yes       /cars/?price=1m-2m&sort=old                     200

Even twelve lines show the problems: the same filters in two orders, a duplicate filter, a Subaru Demio (no such car, still 200), page 48 of a make with fewer pages, and ?make=toyota duplicating /cars/toyota/.

Step 2 · Measure three things per pattern

MeasureHow to read itWhat it suggests
Repeated requests: the same URL fetched again and againA few URLs taking a large share of requests.Something links to them prominently: a menu, sort control or sitemap.
Distinct combinations: different URLs per patternThousands of URLs, each fetched once or twice.Crawlers are exploring an open-ended space.
Share of crawl: the pattern’s part of all verified requestsCompare with its share of clicks or listings.A pattern taking much more crawl than it earns needs a decision.

Then check what the URLs return. A 200 for an empty or nonsensical combination is the first thing Google asks you to fix.

Step 3 · Judge each pattern

Illustrative example, not real data. An invented used-car catalogue, example.co.ke. Seven days of verified Googlebot requests: 84,000 in total, 48,600 of them to /cars/ URLs with parameters.

PatternUser purposeSearch demandCrawl observationsNext investigation
/cars/toyota/ (make as a path)Browse one makeStrong: “toyota cars for sale”2,400 requests, all 200Keep: linked, in the sitemap, indexable.
?makeThe same page as the path versionCovered by the path page6,200 requests; mostly repeats of 30 URLsFind what links to ?make= instead of the path; check its canonical.
?make&modelNarrow to one modelReal: “toyota fielder for sale”7,900 requests, 1,100 URLs; 300 impossible pairs answer 200Consider clean model pages; return 404 for impossible pairs.
?locationCars near the buyerSome: “cars for sale mombasa”1,800 requests, 12 URLsWorth a page only where you have stock in that town.
?priceA budgetSome: “cars under 1 million”2,100 requests, 8 URLsCheck results pages; maybe one guide, not a page per band.
?sort with any filterReorder the same carsNone19,600 requests, 9,400 URLsStrong candidate to keep out of the crawl.
3 or more filters combinedVery narrow browsingAlmost none11,000 requests, 10,200 URLs, many with no resultsReturn 404 for no results; stop crawlable links beyond two filters.

Googlebot requests by URL pattern in one week

  • Matches searches
  • Check: duplicate or thin demand
  • No search demand
/cars/toyota/ (make as a path)2,400 · 3%
?make6,200 · 7%
?make&model7,900 · 9%
?location1,800 · 2%
?price2,100 · 2%
?sort with any filter19,600 · 23%
3 or more filters combined11,000 · 13%

Verified Googlebot requests over seven days to an invented used-car catalogue: 84,000 in total. The percentage is each pattern’s share of the 84,000. Sort and deep combinations together take 30,600 requests (36%).
Source: Illustrative example: invented figures, not real data.

Copy linkDownload SVG
Embed

Result: sort and deep combinations take 30,600 of 84,000 requests (36%) and match no searches. The make, model and location filters do match searches, so they deserve clean URLs rather than a block.

Use it in Ndovu

Crawl Analytics stores each requested path with its parameter names, sorted and without values. /cars/?sort=price_asc&make=toyota is kept as /cars/?make&sort, so patterns group neatly and search terms are never stored. For value-level combinations, read the raw log.

  • Search crawling → Possible crawl inefficiency: the share of search crawler requests With parameters, beside redirects and errors, and a URL requested repeatedly table.
  • Crawl issues: “Crawlers are stuck in an endless set of URLs” (or the one crawler’s name) when one path’s requests nearly all carry a query string, with the crawlers involved and a suggested robots.txt line where one fits.
  • Pages: Filter by URL path (for example /cars/), set the response to Crawlers got errors and sort by Most crawler requests.
  • Activity: a sample of recent requests, filterable by path, Verification, Response and Crawler. Errors and blocks are kept first; totals elsewhere count every request.
  • Pick the data source when you have more than one, and open What this data includes: an origin log can’t see pages served from a CDN or page cache. Why sources disagree: CDN and WordPress logs disagree.

A site audit adds the page-side view: Possible crawl trap (URL explosion), Googlebot spends crawls on parameter duplicates, or Many parameter URLs, kept out of search when links are already nofollow and nothing is indexable. To test a list of URLs or a robots.txt rule first, use the Faceted URL Crawl Risk Analyzer and the robots.txt tester.

From finding to action

What you foundWhat to do next
A filter matches real searchesGive it a clean, linked URL with its own heading, and list it in the sitemap.
A parameter duplicates a path pageLink to the path version; canonicalise the parameter version to it.
Sort, view or session parameters take a large shareKeep them out of the crawl: robots.txt, a fragment, or no crawlable links.
Empty or impossible combinations answer 200Return 404, as Google recommends.
Same filters in different ordersBuild links in one fixed order and stop duplicate filters.
A pattern is blocked but still shows in searchExpected for linked URLs; if it must leave search, allow crawling and use noindex instead.

Check that it worked

  1. Write down each change and its date.
  2. After two to four weeks, compare the same length of period before and after, from the same source.
  3. The pattern’s share of verified search crawler requests should fall, and requests to listings and new pages should rise.
  4. 404s for empty combinations rise at first, then tail off as crawlers stop finding them.
  5. Useful filtered pages appear in Indexing and earn impressions; check with the Indexing report.
When you need to…Read
Read crawler evidenceRead Crawl Analytics
Connect or fix a log sourceConnect your server logs
Explain different counts in different logsCDN and WordPress logs disagree
Fix audit findings in orderPrioritise a technical audit

Sources

Next steps

Was this helpful?