Filters for make, price, location and sort can multiply your URLs. Measure what crawlers request, separate useful filtered pages from the rest, then act.
Your car catalogue lets buyers filter by make, model, price and location, and sort the results. Your logs now show Googlebot asking for URLs like /cars/?sort=price_asc&location=nakuru&price=1m-2m.
Some filtered pages are worth finding in search. Most combinations are not. Measure before you block anything.
The short version. Group crawler requests by parameter pattern. For each pattern, ask whether a searcher would want that page, then compare that with how much crawling it takes. Give useful filters a clean, linked URL; keep the rest out of the crawl; return 404 for empty combinations.
How a few filters become thousands of URLs
Each optional filter multiplies the others. With 30 makes, 8 price bands, 12 locations and 4 sort orders, /cars/ alone can produce (30+1) × (8+1) × (12+1) × (4+1) = 18,135 URLs. Add models, page numbers and parameters in a different order, and the count keeps growing.
Google names two costs: crawlers request a very large number of these URLs, and the time spent on them leaves less for new, useful pages (Google).
What Google recommends
| If the filtered URLs… | Google’s options |
|---|---|
| Don’t need to be in search | Disallow them in robots.txt; or put filters after a # fragment, which Google generally ignores; or point them to the unfiltered page with rel="canonical" (may reduce their crawling over time, less effective long term); or add rel="nofollow" to every link to them. |
| Should be crawled and indexed | Use & between parameters, keep filters in a consistent order, prevent duplicate filters, and return a 404 for combinations with no results, duplicate filters, nonsense values or pages past the last one. |
Source for both rows: Google’s faceted navigation guide. Two side effects to plan for: robots.txt manages crawling and does not keep a URL out of Google, and a blocked URL linked from elsewhere can still appear without a description (Google). And a noindex only works if the page is not blocked in robots.txt (Google), so don’t combine the two on one pattern.
Step 1 · Collect the requests
Use verified search crawler requests from your server or CDN logs, at least a week of them. A small sample looks like this:
time crawler verified path status
2026-11-10 02:14 Googlebot yes /cars/toyota/ 200
2026-11-10 02:15 Googlebot yes /cars/?make=toyota 200
2026-11-10 02:15 Googlebot yes /cars/?make=toyota&sort=price_asc 200
2026-11-10 02:16 Googlebot yes /cars/?sort=price_asc&make=toyota 200
2026-11-10 03:40 Googlebot yes /cars/?location=mombasa 200
2026-11-10 03:41 Googlebot yes /cars/?location=mombasa&price=500k-1m&sort=new 200
2026-11-10 05:02 Googlebot yes /cars/?make=subaru&model=demio 200
2026-11-10 05:03 Googlebot yes /cars/?make=toyota&make=toyota 200
2026-11-10 06:20 Googlebot yes /cars/?make=toyota&page=48 200
2026-11-11 02:14 Googlebot yes /cars/?make=toyota 200
2026-11-11 02:16 Googlebot yes /cars/?make=toyota&sort=price_asc 200
2026-11-11 09:12 Bingbot yes /cars/?price=1m-2m&sort=old 200
Even twelve lines show the problems: the same filters in two orders, a duplicate filter, a Subaru Demio (no such car, still 200), page 48 of a make with fewer pages, and ?make=toyota duplicating /cars/toyota/.
Step 2 · Measure three things per pattern
| Measure | How to read it | What it suggests |
|---|---|---|
| Repeated requests: the same URL fetched again and again | A few URLs taking a large share of requests. | Something links to them prominently: a menu, sort control or sitemap. |
| Distinct combinations: different URLs per pattern | Thousands of URLs, each fetched once or twice. | Crawlers are exploring an open-ended space. |
| Share of crawl: the pattern’s part of all verified requests | Compare with its share of clicks or listings. | A pattern taking much more crawl than it earns needs a decision. |
Then check what the URLs return. A 200 for an empty or nonsensical combination is the first thing Google asks you to fix.
Step 3 · Judge each pattern
Illustrative example, not real data. An invented used-car catalogue, example.co.ke. Seven days of verified Googlebot requests: 84,000 in total, 48,600 of them to /cars/ URLs with parameters.
| Pattern | User purpose | Search demand | Crawl observations | Next investigation |
|---|---|---|---|---|
/cars/toyota/ (make as a path) | Browse one make | Strong: “toyota cars for sale” | 2,400 requests, all 200 | Keep: linked, in the sitemap, indexable. |
?make | The same page as the path version | Covered by the path page | 6,200 requests; mostly repeats of 30 URLs | Find what links to ?make= instead of the path; check its canonical. |
?make&model | Narrow to one model | Real: “toyota fielder for sale” | 7,900 requests, 1,100 URLs; 300 impossible pairs answer 200 | Consider clean model pages; return 404 for impossible pairs. |
?location | Cars near the buyer | Some: “cars for sale mombasa” | 1,800 requests, 12 URLs | Worth a page only where you have stock in that town. |
?price | A budget | Some: “cars under 1 million” | 2,100 requests, 8 URLs | Check results pages; maybe one guide, not a page per band. |
?sort with any filter | Reorder the same cars | None | 19,600 requests, 9,400 URLs | Strong candidate to keep out of the crawl. |
| 3 or more filters combined | Very narrow browsing | Almost none | 11,000 requests, 10,200 URLs, many with no results | Return 404 for no results; stop crawlable links beyond two filters. |
Googlebot requests by URL pattern in one week
- Matches searches
- Check: duplicate or thin demand
- No search demand
Verified Googlebot requests over seven days to an invented used-car catalogue: 84,000 in total. The percentage is each pattern’s share of the 84,000. Sort and deep combinations together take 30,600 requests (36%).
Source: Illustrative example: invented figures, not real data.
Result: sort and deep combinations take 30,600 of 84,000 requests (36%) and match no searches. The make, model and location filters do match searches, so they deserve clean URLs rather than a block.
Use it in Ndovu
Crawl Analytics stores each requested path with its parameter names, sorted and without values. /cars/?sort=price_asc&make=toyota is kept as /cars/?make&sort, so patterns group neatly and search terms are never stored. For value-level combinations, read the raw log.
- Search crawling → Possible crawl inefficiency: the share of search crawler requests With parameters, beside redirects and errors, and a URL requested repeatedly table.
- Crawl issues: “Crawlers are stuck in an endless set of URLs” (or the one crawler’s name) when one path’s requests nearly all carry a query string, with the crawlers involved and a suggested robots.txt line where one fits.
- Pages: Filter by URL path (for example
/cars/), set the response to Crawlers got errors and sort by Most crawler requests. - Activity: a sample of recent requests, filterable by path, Verification, Response and Crawler. Errors and blocks are kept first; totals elsewhere count every request.
- Pick the data source when you have more than one, and open What this data includes: an origin log can’t see pages served from a CDN or page cache. Why sources disagree: CDN and WordPress logs disagree.
A site audit adds the page-side view: Possible crawl trap (URL explosion), Googlebot spends crawls on parameter duplicates, or Many parameter URLs, kept out of search when links are already nofollow and nothing is indexable. To test a list of URLs or a robots.txt rule first, use the Faceted URL Crawl Risk Analyzer and the robots.txt tester.
From finding to action
| What you found | What to do next |
|---|---|
| A filter matches real searches | Give it a clean, linked URL with its own heading, and list it in the sitemap. |
| A parameter duplicates a path page | Link to the path version; canonicalise the parameter version to it. |
| Sort, view or session parameters take a large share | Keep them out of the crawl: robots.txt, a fragment, or no crawlable links. |
| Empty or impossible combinations answer 200 | Return 404, as Google recommends. |
| Same filters in different orders | Build links in one fixed order and stop duplicate filters. |
| A pattern is blocked but still shows in search | Expected for linked URLs; if it must leave search, allow crawling and use noindex instead. |
Check that it worked
- Write down each change and its date.
- After two to four weeks, compare the same length of period before and after, from the same source.
- The pattern’s share of verified search crawler requests should fall, and requests to listings and new pages should rise.
- 404s for empty combinations rise at first, then tail off as crawlers stop finding them.
- Useful filtered pages appear in Indexing and earn impressions; check with the Indexing report.
| When you need to… | Read |
|---|---|
| Read crawler evidence | Read Crawl Analytics |
| Connect or fix a log source | Connect your server logs |
| Explain different counts in different logs | CDN and WordPress logs disagree |
| Fix audit findings in order | Prioritise a technical audit |
Sources
- Google: Managing crawling of faceted navigation URLs
- Google: Introduction to robots.txt
- Google: Block search indexing with noindex