NDOVUBOT · OUR CRAWLER
Meet NdovuBot,
the crawler behind our audits.
NdovuBot fetches pages only when a NdovuSEOLabs user audits a website, checks its sitemap or tracking, or adds it as a project. It follows robots.txt, slows down when asked and signs every request so you can verify it is really us.
- Follows robots.txt
- Signed with Web Bot Auth
- About one request a second
- User agentNdovuBot/1.0With a link to this page
- Signature-Agentapp.ndovuseolabs.comEd25519, Web Bot Auth
- robots.txtNdovuBotOpenSEO-Audit rules still apply
- PaceAbout 1 request a secondBacks off on 429
BOT DETAILS
Everything a site owner needs, in one table.
Last reviewed 29 October 2026.
| Bot name | NdovuBot |
|---|---|
| User agent | NdovuBot/1.0 (+https://ndovuseolabs.com/ndovubot/) |
| robots.txt token | NdovuBot (earlier rules for OpenSEO-Audit still apply) |
| Verification | Web Bot Auth signature, Signature-Agent: "https://app.ndovuseolabs.com" |
| Public key | app.ndovuseolabs.com/.well-known/http-message-signatures-directory |
| IP addresses | None fixed: it runs on Cloudflare Workers, so verify the signature instead |
| Operator | Ndovu SEO Labs, Kenya |
| Purpose | SEO site audits, sitemap and tracking checks, and reading a site when it is added as a project |
| Contact | Contact the team about crawl problems or abuse |
WHAT IT NEVER DOES
A crawler that reads, and nothing else.
NdovuBot only sends GET and HEAD requests to public pages.
- It never submits forms, posts comments or adds items to a cart.
- It never tries to log in, guess passwords or get past a paywall or login page.
- It never crawls on its own schedule: every visit starts from a person auditing or checking that site.
- It never collects content to train AI models.
WHEN IT VISITS
It comes when someone asks it to.
NdovuBot does not crawl the web on its own. Every visit starts from a person working on a website in NdovuSEOLabs.
Site audits
Reads pages to find broken links, missing titles, redirect chains and other issues, up to the page limit the user sets.
Sitemap and tracking checks
Fetches robots.txt, sitemaps and a page or two to confirm Search Console sitemaps and analytics tags.
New projects
Reads up to five pages when a website is added, so the project starts with a short summary of what the site does.
HOW IT IDENTIFIES ITSELF
A name you can read, a signature you can check.
A user agent can be copied by anyone. A signature cannot.
Every request sends this user agent:
NdovuBot/1.0 (+https://ndovuseolabs.com/ndovubot/)Requests from app.ndovuseolabs.com are also signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). Each request carries Signature, Signature-Input and Signature-Agent: "https://app.ndovuseolabs.com" headers, signed with our Ed25519 key. The public key is published here:
https://app.ndovuseolabs.com/.well-known/http-message-signatures-directoryNdovuBot runs on Cloudflare Workers, so it has no fixed IP addresses. Verify the signature instead of an IP list.
ROBOTS.TXT
Tell it where it may go.
NdovuBot reads robots.txt before crawling and skips every page it disallows.
Rules for NdovuBot take priority over rules for *. To keep it out of your whole site:
User-agent: NdovuBot
Disallow: /NdovuBot used to be called OpenSEO-Audit. If your robots.txt has rules for that name and none for NdovuBot, they still apply.
Check which rule applies to a URL with the free Robots.txt URL Tester: enter NdovuBot as the user-agent token.
VERIFY A REQUEST
Check it is really NdovuBot.
Three ways, from least to most effort.
- On Cloudflare: sites using Bot Management or Web Bot Auth verification check the signature automatically against the key above.
- Anywhere else: fetch the key directory, then verify the request’s
Signatureheader over@authorityandSignature-Agentwith any Web Bot Auth library. - Quick check: a request that names NdovuBot but has no
Signature-Agentheader did not come from app.ndovuseolabs.com.
FIREWALLS AND RATE LIMITS
Let it through, or slow it down.
NdovuBot sends about one request a second and fetches only as many pages as the person running the audit chose.
- When a site answers
429 Too Many Requests, NdovuBot slows down and follows theRetry-Afterheader. If the site keeps asking it to wait, the audit stops early. - To allow it through bot protection, skip the rule for requests with a verified signature, or for user agents containing
NdovuBot. On Cloudflare, a WAF custom rule can do either. - On Shopify, use Crawler access (Online Store → Preferences → Crawler access) and paste the signature into NdovuSEOLabs under Settings → Crawler access.
FAQ
About NdovuBot.
What it does, what it does not do, and how to control it.
Does NdovuBot collect content to train AI?
No. It reads pages to report on them in the audit the user ran. It does not index the web or collect training data.
Why does my log still show OpenSEO-Audit?
That was NdovuBot’s earlier name. Requests before 29 October 2026 used it, and robots.txt rules written for it are still honoured.
Are all audit requests made by NdovuBot?
Almost. Two optional audit features load pages another way: “Render JavaScript” uses a headless browser (usually Cloudflare Browser Run), and Lighthouse checks run from DataForSEO’s servers with Lighthouse’s own Chrome user agent. Neither sends the NdovuBot user agent or a signature.
Is NdovuBot the spider in your footer?
The footer bot is its mascot: it crawls this website’s links and documents every page, right in your browser. The real NdovuBot only visits sites when someone audits them.
Seeing NdovuBot in your logs?
If it is causing trouble, tell us the site and the time, and we will look at it.