Skip to main content

Try Base free for 14 days, no card or M-Pesa needed See pricing

Guide

Which bots use the most bandwidth on Kenyan websites?

Last updated
Page tools
Agent setup

Ndovu research on 14 Kenyan websites: which crawlers received the most data, why requests and data rank bots differently, and how much went on errors.

Across 14 Kenyan websites, bots received 250.1 GB of responses in four days. Applebot, GPTBot and Meta-ExternalAgent received the most: 22%, 21% and 13% of it.

But the typical website saw a different picture. Applebot’s share on the median website was 0.27%, and a crawler’s request count is a poor guide to how much data it takes.

Key findings

  1. 82.1%of all bot data went to three websites.
  2. 8th → 5thGooglebot’s rank by requests and by data; its average response was 292 KB.
  3. 221 KBthe average “not found” page, more than a successful response (158 KB).
  4. 13.8 GB vs 0.3 GBbot data a day for the median content site and the median store.

Third in Ndovu’s bot research series, after which bots crawl Kenyan websites and how bots crawl ecommerce websites.

What “bandwidth” means here

We measure the response size recorded in each website’s server access log: the size field of the standard log format, which Apache defines as the response size without headers, not the exact bytes sent over the network (Apache). Redirects and 304 “not modified” responses record no body, so they count as zero.

  • Origin only. 14 websites with server logs, 56 website-days. Requests a CDN answers from its cache never reach the server, so they are not here.
  • Bots only. All crawlers and automated agents except our own NdovuBot. 85.3% of the data went to verified crawlers.
  • Not a bill. Hosting and CDN plans meter differently and include allowances, so this does not convert to shillings.

Which bots received the most data

BotDataShare of dataShare of requestsAverage response
Applebot54.9 GB22.0%23.3%141 KB
GPTBot51.9 GB20.7%20.9%149 KB
Meta-ExternalAgent32.9 GB13.1%11.0%178 KB
Amazonbot15.0 GB6.0%7.5%120 KB
Googlebot11.7 GB4.7%2.4%292 KB
ClaudeBot10.5 GB4.2%3.3%187 KB
ShapBot7.5 GB3.0%2.6%172 KB
Bingbot5.3 GB2.1%1.7%184 KB
AhrefsBot5.1 GB2.1%0.9%343 KB
Meta-ExternalAds5.1 GB2.0%5.4%56 KB

Table 1. The ten named bots that received the most recorded data across 14 websites. Shares use all 250.1 GB and all 1,669,357 external requests, including unnamed agents (“other self-declared bots” took 6.6% of data). Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

More requests do not mean more data

More requests do not mean more data

  • Share of requests n=1,669,357
  • Share of data n=250.1 GB
Googlebotaverage response 292 KB2.4% · 4.7%
AhrefsBotaverage response 343 KB0.9% · 2.1%
Meta-ExternalAgentaverage response 178 KB11.0% · 13.1%
Amazonbotaverage response 120 KB7.5% · 6.0%
Meta-ExternalAdsaverage response 56 KB5.4% · 2.0%

Bots that fetch large pages take more data than their request count suggests; bots that fetch small files take less. Data is the access log’s response size (body, no headers).
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Copy linkDownload SVG
Embed

Googlebot’s average response was 292 KB and AhrefsBot’s 343 KB, against 150 KB for all bots. Meta-ExternalAds went the other way: 5.4% of requests but 2.0% of data, at 56 KB a response. To find the bot behind a bandwidth rise, rank by data, then look at what it fetched.

The busiest bot overall is rarely the busiest on your site

Share of all bot data vs average share per website

  • Share of all bot data n=250.1 GB
  • Average share per website n=14 websites
Applebot22.0% · 5.2%
GPTBot20.7% · 12.9%
Meta-ExternalAgent13.1% · 12.9%
Googlebot4.7% · 8.2%

The average per website gives each website one vote, with zero where a bot was absent.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Copy linkDownload SVG
Embed

86% of Applebot’s data went to one website. Give every website equal weight and its share falls to 5.2%; GPTBot (12.9%) and Meta-ExternalAgent (12.9%) lead instead, and Googlebot rises to 8.2%.

Bot data is uneven between websites too: the mean website received 4.5 GB a day, the median 0.7 GB.

Content sites and stores

WebsitesBot data per site per day (median)MeanBiggest bot on the average site
Content and tools313.8 GB12.8 GBGPTBot (47%)
Ecommerce80.3 GB2.3 GBMeta-ExternalAgent (19%)

Table 2. Bot data by sector, server-log websites only. Real estate and services have too few comparable websites. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Content and tools websites drew far more bot data per website, led by GPTBot. On stores, the data followed the page type:

Store page typeShare of page dataAverage response
Generated comparison pages38%413 KB
Product pages38%385 KB
Category and shop pages13%134 KB
Images6%85 KB
Homepage1%81 KB
Scripts, styles and fonts1%34 KB
API and AJAX0%5 KB

Table 3. Recorded data by page type on 8 stores, named bots only. Comparison pages are a generated URL family on two stores; see the ecommerce study. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

How much went on errors

ResponseRequestsDataAverage response
Success (2xx)1,554,129245.5 GB158 KB
Redirect (3xx)53,9490.0 GB0 KB
Not found (404, 410)18,0374.0 GB221 KB
Forbidden (403)3,1190.6 GB191 KB
Too many requests (429)31,9800.0 GB0 KB
Server error (5xx)4,3820.0 GB0 KB

Table 4. Bot responses by status across 14 websites. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Errors were 3.5% of requests and 1.8% of data, about 4.6 GB. Small overall, but two details matter:

  • A 404 page is not small. At 221 KB it averaged more than a successful response, a size that means full themed pages, not short messages. A lighter 404 template cuts the cost of every bad link a bot follows.
  • Refusing cheaply works. 429 and 5xx responses averaged under 1 KB; 403s averaged 191 KB, a full page again. If you block a bot, block it before the theme renders.
  • One name stood out on stores. Bytespider, almost all of whose requests failed, averaged 12.0% of a store’s bot data (median 0.2%): heavy on a few small stores, slight on the rest.

Redirects record no body but each one costs a request; a chain or loop multiplies that, and only a traced chain of targets proves a loop. Filtered and generated URLs cost data through successful responses, so they never show up as errors. For Google, 429 and 5xx slow crawling down (Google), and empty filter combinations should return a 404 (Google).

Find the bot behind your bandwidth

StepWhat to do
1. Know the measureCheck whether the bytes come from the CDN, the server log or WordPress, and the period. Don’t mix them.
2. Verify the botSeparate verified crawlers from names in the user-agent (Google’s method).
3. Rank by data, then by pageFor the top bots, group bytes by section, status and file type: many pages, a few large files, or error pages?
4. Fix the sourceRemove stale links, shorten redirect chains, stop generating needless URLs, lighten 404 and 403 pages.
5. Measure againSame source, same URL groups, a comparable period. Check important pages still reach the crawlers you want.

Ndovu’s Crawl Analytics shows requests and data by bot, response and page from your own logs.

Method and limitations

How it works
Sample14 Kenyan websites with server access logs monitored by NdovuSEOLabs; not a random sample. Plugin-only websites excluded.
Period14–17 November 2026: four complete days in Kenyan time. A pilot; later editions use 28 days.
DataThe access log’s response-size field: body size without headers, not bytes on the wire. Exact sums, not rounded dashboard values. GB = 10⁹ bytes, KB = 10³.
SharesShare of data: a bot’s bytes ÷ all bot bytes. Average and median: the share on each website, then the mean or median, with zero where a bot was absent.
Page typesStores only, by WooCommerce’s default URL patterns, named bots only.
Not measuredCDN traffic, human traffic, total website bandwidth, hosting cost and long-term trends.

Original research by NdovuSEOLabs, from Crawl Analytics data. You may cite these figures with a link to this page.

Next steps

Was this helpful?