Skip to main content

Try Base free for 14 days, no card or M-Pesa needed See pricing

Guide

How much crawler bandwidth goes to error responses?

Last updated
Page tools
Agent setup

Ndovu research on 14 Kenyan websites: how much bot data went on 404, 403, 429 and 5xx responses, which crawlers received it, and why stores were hit hardest.

Across 14 Kenyan websites, error responses were 3.5% of bot requests and 1.8% of the data bots received: about 4.6 GB in four days. That small total hides a big spread. On the average website errors took 12.8% of bot data, and on 4 of 8 online stores they took more than a fifth.

Key findings

  1. 32% → 68%one website’s error share of requests against its error share of data: an error rate is not an error bandwidth.
  2. 21% vs 0.4%of bot data went on errors on the average store and the average content site.
  3. 40%of all error data went to Bytespider, and every byte it received was an error.
  4. 81%of error requests came from verified crawlers: most errors reached real search and AI bots.

Fourth in Ndovu’s bot research series. The sample and method are in which bots crawl Kenyan websites; total bot data is in which bots use the most bandwidth.

What counts as an error

ResponseIn the error figures?Why
404, 410YesThe page is missing or removed.
403, 401, 429Yes, reported apartRefused or rate-limited: may be on purpose.
500–599YesThe server failed.
301, 302, 307, 308No, reported apartRedirects; they record no body here.
304NoA cache check; it cannot carry content (RFC 9110).
200 showing an error pageNot visibleA “soft 404”; Google finds these from the content (Google), status codes alone cannot.

Error data is the response size each server logged for those responses: exact sums per status, never estimated from an error rate. Origin server logs only; requests a CDN answers never reach them.

An error rate is not an error bandwidth

Multiplying an error rate by total bandwidth assumes error pages are the same size as everything else. They rarely are:

ResponseRequestsDataAverage response
Success (2xx)1,554,129245.5 GB158 KB
Not found (404, 410)18,0374.0 GB221 KB
Forbidden (403)3,1190.6 GB191 KB
Too many requests (429)31,9800.0 GB0.3 KB
Server error (5xx)4,3820.0 GB0.4 KB

Table 1. Bot responses by status across 14 websites. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

A 404 averaged 221 KB, more than a success. A 429 averaged 0.3 KB. So a website whose errors are mostly 404s loses more data than its error rate suggests, and one whose errors are mostly 429s loses almost none.

Errors: share of requests vs share of data, by website

  • Errors, share of requests n=14 websites
  • Errors, share of data
1. Store32.3% · 68.5%
2. Store47.0% · 57.1%
3. Store29.5% · 22.5%
4. Store13.7% · 20.6%
5. Other website4.6% · 5.5%
6. Store5.9% · 1.9%
7. Other website1.6% · 1.2%
8. Other website3.6% · 1.0%

The eight websites with the highest share of bot data on errors, unnamed; the other 6 were under 0.8%. Errors are 403, 404, 410, 429, other 4xx and 5xx.
Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

Copy linkDownload SVG
Embed

Stores were hit hardest

WebsitesError share of bot data, averageMedianError data per site per day, median
All websites1412.8%1.1%13 MB
Ecommerce821.4%11.3%24 MB
Content and tools30.36%0.03%7 MB

Table 2. Each website’s error share of bot data, then the average and median across websites. Pooled across all websites the share is 1.8%, because the busiest websites had few errors. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

On stores, the failed requests were mostly images (6,752) and product pages (4,644), then generated comparison and category pages: most likely URLs that no longer exist but are still linked or remembered. On a small store with modest bot traffic, those full error pages can outweigh the real pages bots fetch. Content sites barely registered.

Which bots received the error data

BotError dataShare of all error dataErrors, share of its requestsErrors, share of its data
Bytespider1,821 MB39.6%99.1%100.0%
Meta-ExternalAgent1,180 MB25.7%10.1%3.6%
Googlebot276 MB6.0%2.2%2.4%
Bingbot224 MB4.9%2.6%4.3%
PetalBot198 MB4.3%3.9%4.9%
Meta-WebIndexer140 MB3.1%3.3%5.3%

Table 3. The six bots that received the most error data. Source: NdovuSEOLabs Crawl Analytics, 14–17 November 2026.

  • Bytespider got errors on 99.1% of its requests across 12 websites; none could be verified as ByteDance. Blocking it at the server or CDN, before the theme renders, would remove most of this data.
  • Meta-ExternalAgent received 26% of error data, almost all of it 404 pages; its many 429s cost almost nothing.
  • Storebot-Google spent 32% of its data on errors, almost all 404s for missing scripts and styles on stores.

What each error type tells you

StatusWhat we sawWhat it means
40418,037 requests, 4.0 GB, the bulk of error data.Bots still find removed URLs. Fix the links and sitemaps that list them; keep the 404.
4033,119 requests, 89% to Bytespider.Deliberate blocks, but served as full pages. Block earlier and cheaper.
42931,980 requests: GPTBot 17,006 on 9 websites, Meta-ExternalAgent 12,236 on 10.The server rate-limiting busy AI crawlers. Cheap in data, but Google reads 429 as overload (Google); check Googlebot is never limited.
5xx4,382 requests, 88% from Baiduspider on 1 website, with empty bodies.A server failing for one crawler. Little data, but a real fault to fix.

Google advises against using 401 or 403 to slow crawling; other 4xx responses do not change its crawl rate (Google).

Bots that received nothing but errors

Of 539 bot-and-website pairs, 34 got only errors: 9,949 requests on 11 websites, still 0.7 GB of data. Two names made up 99% of those requests: Bytespider (5,974) and Baiduspider (3,856).

A bot that never gets a useful page is a candidate for a robots.txt rule or a block, if you do not want it. If you do, the errors are the thing to fix.

Measure it on your own website

DoAvoid
Sum the bytes logged against 4xx and 5xx responses, divided by all bot bytes, from the same log and period.Error rate × total bandwidth.
Report error data per day as well as the share; a share can rise while the data falls.Reading a falling share as an improvement on its own.
Treat zero bytes as what they are: redirects, 304s and many 429s and 5xx have no body.Treating a zero as missing data, or missing data as zero.
Use full totals. Ndovu’s Activity view samples errors first, so it overstates them.Computing shares from a sample of recent requests.
Keep CDN and origin figures apart.Adding edge and origin bytes for the same request.

What to fix first

  • Errors on pages that should work. A product or category returning an error costs visibility, whatever its size.
  • Old links that keep producing 404s. Find where bots discover them (menus, sitemaps, feeds, other sites) and update the source.
  • Heavy error pages. A lighter 404 template, and blocks applied at the server or CDN instead of inside WordPress.
  • Filters that create empty pages. Return a 404 for empty combinations (Google) and stop linking them; see filtered URL crawling.

Never “fix” an error rate by returning 200 for missing pages; that hides the problem and creates soft 404s. Measure the same URLs and source again after each change. Ndovu’s Crawl Analytics breaks bot traffic down by response, bot and page.

Method and limitations

How it works
Sample14 Kenyan websites with server access logs, monitored by NdovuSEOLabs; not a random sample.
Period14–17 November 2026: four complete days in Kenyan time. A pilot; later editions use 28 days.
Errors403, 404, 410, 429, other 4xx and 5xx. Redirects, 304s and soft 404s are not counted.
DataThe access log’s response-size field (body without headers), summed per status. GB = 10⁹ bytes.
SharesPooled: error bytes ÷ all bot bytes. Average and median: each website’s share first. NdovuBot excluded; unverified agents included.
Not measuredWhich URLs failed, why bots requested them, CDN responses, hosting cost and long-term trends.

Original research by NdovuSEOLabs, from Crawl Analytics data. You may cite these figures with a link to this page.

Next steps

Was this helpful?