Skip to main content

Try Base free for 14 days, no card or M-Pesa needed See pricing

Guide

How bots crawl ecommerce websites in Kenya

Last updated
Page tools
Agent setup

Ndovu research on eight Kenyan online stores: which bots crawled them, which pages they requested, and how filters, generated pages and errors used the crawls.

Eight Kenyan online stores recorded 332,783 bot requests in four days. The average store received 10,399 a day; the median store, 3,365. Most of that activity went to a few stores, and much of it went to filtered category pages and generated URLs rather than products.

Key findings

  1. 50.2%of all store bot requests went to one store; two stores took 75.4%.
  2. Meta-ExternalAgentwas the busiest named bot; Googlebot and Bingbot reached every store with verified requests.
  3. 88.7%of category-page requests carried filters, sorting or paging in the query string.
  4. 46,938generated comparison-page addresses drew 20.6% of page requests in four days.
  5. 98.7%of requests calling themselves Bytespider got an error.

This is the second article in Ndovu’s bot research series. The method and the wider sample are in Which bots crawl Kenyan websites?.

The stores in this study

11 online stores had data. The 8 with server access logs for 14–17 November 2026 are compared here: 32 store-days. The 3 with WordPress plugin data only see fewer requests, so they are left out. Most sell consumer electronics, computers, accessories or specialist equipment. Our own NdovuBot (3,123 requests) is excluded, and no store is named.

A few stores get most of the bot traffic

Bot requests per store per day

  • Requests per day n=8 stores
  • Mean 10,399
  • Median 3,365
Store 141,802
Store 220,932
Store 311,068
Store 43,593
Store 53,137
Store 62,135
Store 7294
Store 8235

Each store’s four-day total ÷ 4, stores sorted and unnamed. The two busiest pull the mean far above the median. NdovuBot excluded.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Copy linkDownload SVG
Embed

The busiest store received 50.2% of all requests. The mean of 10,399 a day describes almost no store; the median of 3,365 is closer to a typical shop. Neither is a target: request counts only mean something next to the size of the catalogue and the responses returned.

Which bots crawled the stores

BotRequestsShareStores reached / verifiedVerified
Meta-ExternalAgent85,37525.65%7 / 7>99.9%
Meta-ExternalAds42,40612.74%4 / 4100.0%
ShapBot32,1689.67%2 / –No check
Googlebot30,5019.17%8 / 899.4%
AhrefsBot12,8313.86%8 / 892.7%
GPTBot12,8203.85%7 / 699.9%
Bingbot11,3833.42%8 / 899.8%
Meta-WebIndexer10,1393.05%7 / 699.8%
PetalBot8,8372.66%6 / 699.5%
RiverBot7,4182.23%2 / 299.8%

Table 1. The ten named bots with the most requests across 8 stores. Shares use all 332,783 requests. “Verified” means the request matched the operator’s published IP ranges or DNS (how Google describes it); “No check” means the operator publishes nothing to check a server log against. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

81.6% of all store requests were verified. Meta’s two agents together made 38.4% of requests. ShapBot’s large total came from 2 stores only.

What a typical store sees

Share of all store requests vs average share per store

  • Share of all requests n=332,783 requests
  • Average share per store n=8 stores
Meta-ExternalAgent25.7% · 17.1%
Googlebot9.2% · 11.2%
Meta-ExternalAds12.7% · 6.4%
Bytespider1.9% · 7.9%
GPTBot3.9% · 3.3%
ShapBot9.7% · 2.6%

The average per store gives each store one vote, with zero where a bot was absent.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Copy linkDownload SVG
Embed

Meta-ExternalAgent still leads at 17.14% per store (median 12.35%), with Googlebot next at 11.21%. Bytespider moves the other way: 1.9% of all requests but 7.9% on the average store, because it is a large share of traffic on small stores. Meta-ExternalAds is the reverse: 12.7% overall, but a median store share of 0.25%.

Google is more than the Googlebot row

Google agentRequestsStores with verified requestsMostly requested
Googlebot30,5018 of 8generated comparison pages, product pages
Googlebot-Image3,5118 of 8images, product pages
AdsBot-Google3,4226 of 8scripts, styles and fonts, API and AJAX
Storebot-Google3,2224 of 8scripts, styles and fonts, images

Table 2. Google’s agents on the 8 stores. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Storebot-Google checks shopping details such as price, availability and shipping, and Google says it may walk through checkout without buying (Google Merchant Center). In this window it requested scripts, images and API calls, and no cart or checkout pages. If a checkout appears in your logs without an order, verify the agent before blocking it.

Which pages the bots requested

Where bots went on stores: requests vs data, by page type

  • Share of requests n=323,641
  • Share of recorded data n=72.2 GB
Product pages21.8% · 37.6%
Category and shop pages21.4% · 12.9%
Generated comparison pages20.6% · 38.2%
Images15.0% · 5.7%
API and AJAX6.5% · 0.1%
Scripts, styles and fonts5.6% · 0.9%
Homepage3.4% · 1.3%
Other pages2.8% · 1.7%

Named bots only. Page types follow WooCommerce’s default URLs; comparison pages are a generated URL family on two stores.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Copy linkDownload SVG
Embed
Page typeRequestsDistinct pathsWith parametersRedirectedErrors
Product pages70,58238,29129%15%7%
Category and shop pages69,4015,80589%4%3%
Generated comparison pages66,71446,9386%25%4%
Images48,39437,9700%0%14%
API and AJAX21,1491,0166%0%4%
Scripts, styles and fonts18,2515,11694%0%11%
Homepage11,08412174%19%1%
Other pages9,0993,0775%66%16%
Sitemaps, feeds and robots.txt4,0963,1440%3%4%
Cart, checkout and account3,4525185%14%5%

Table 3. Requests from named bots by page type (323,641 of the 332,783; spoofed, unknown and generic agents are not kept per page). Distinct paths ignore query values, which Ndovu never stores. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Bots split their effort very differently:

  • Meta-ExternalAgent sent 48% of its requests to category pages and 17% to products.
  • Googlebot sent 35% to products, 37% to comparison pages and 4% to categories.
  • GPTBot sent 54% to comparison pages; OAI-SearchBot mostly fetched images (57%) and the homepage.
  • Bingbot was the most even: 30% products, 26% images.

These are combined across the stores, so a bot’s split leans towards the stores where it was busiest.

Filtered category pages

88.7% of category requests carried a query string, against 29.3% for products. In 5 of 8 stores most category requests did. On the store where it was most extreme, category pages were 62% of all bot requests, 100% of them with parameters.

Filters multiply quickly: four filters with 10, 8, 6 and 5 options allow 2,400 combinations before sorting and paging. Google warns that this can cause overcrawling and slower discovery, and recommends a 404 for combinations with no results (Google). Decide which filtered pages deserve to be found, give those stable links, and keep the rest from producing new URLs. Our guide to investigating filtered URL crawling shows how to measure it.

Generated pages can outgrow the catalogue

Product-comparison pages, found on 2 stores, drew 66,714 requests across 46,938 different addresses in four days, more addresses than all product pages (38,291). They took 38.2% of recorded page data, and 25% were redirected.

A page made for every pair of products grows with the square of the catalogue. If those pages are useful in search, link the ones that matter; if not, stop linking every combination and check what the redirects point to. Either way, measure the same URL family again after the change.

Errors and the Bytespider pattern

Error responses were 6.9% of store requests and 5.0% of recorded data, higher than across the whole study. 14% of image requests and 11% of script and style requests failed, so look for removed product images and old theme files that are still linked.

Requests calling themselves Bytespider reached all 8 stores. 6,142 of 6,222 failed: 5,122 “not found” and 1,018 “forbidden”. 60% were for images. ByteDance publishes nothing to check them against, so this describes the name, not the company.

If the errors are…CheckThen
Current products or imagesThe exact response, and any security or cache layer in frontRestore access for the crawlers you want
Removed productsInternal links, sitemaps and feeds that still list themRemove the old references; a 404 or 410 is correct
Invalid filter combinationsWhich links generate themFix the links; keep the 404
Redirects that never settleThe chain of redirect targetsPoint each URL straight to its final address

Where the data went

The stores’ access logs recorded 73.8 GB of responses to bots in four days. Product pages took 38% of recorded page data, generated comparison pages 38% and images only 6%. This is the response body the origin server logged; requests answered by a CDN never reach it, and it cannot be turned into a hosting cost without the bill.

Make sure bots can reach the catalogue

CheckWhySource
Menus link to categories, categories link to every product you want indexedGooglebot generally does not use your search box to find productsGoogle
Later category pages have real links, not only a “Load more” buttonGoogle’s crawlers do not click buttons; they follow <a href> linksGoogle
Each product has one main URL; variant, tracking and filter links do not create copiesDuplicate addresses split and waste crawlingVariants guide
New and changed products get requested soon after you publish themProves discovery, not just activityCrawl Analytics

Most stores do not have a crawl-budget problem: Google’s guide on it is for sites with over a million pages, or over 10,000 changing daily (Google). The problems visible in these stores were smaller and fixable: crawls spent on filters, generated pages and dead links instead of products. And a request is not a result: a GPTBot visit does not put a product in ChatGPT, and a Googlebot visit does not index it (OpenAI).

Method and limitations

How it works
Sample8 Kenyan online stores with server access logs, monitored by NdovuSEOLabs; not a random sample. 3 plugin-only stores excluded.
Period14–17 November 2026: four complete days in Kenyan time, 32 store-days. A pilot; later editions use 28 days.
Requests and sharesAll automated requests except NdovuBot. Daily rates are each store’s total ÷ 4, then the mean or median. Average and median shares weight each store equally, with zero where a bot was absent.
VerificationExact counts of requests matching the operator’s published IP ranges or reverse DNS. “No check” where none is published.
Page typesEach path classified by WooCommerce’s default URL patterns, inside the database. Named bots only. Query values are never stored, so distinct paths undercount filter combinations.
DataResponse body size from the access log, headers excluded. GB = 10⁹ bytes.
Not measuredCheckout steps completed, indexing, rankings, sales, hosting cost, and long-term trends.

Original research by NdovuSEOLabs, from Crawl Analytics data. You may cite these figures with a link to this page.

Next steps

Was this helpful?