OTHER AGENTS · UNKNOWN
Scripts and headless browsers
General-purpose tools: these could be anyone, including scrapers and AI agents that hide their name.
- robots.txt behaviour not documented
- Cannot be verified
- No public documentation
- UnverifiedIdentityNo published check
- Unknownrobots.txtrobots.txt behaviour not documented
- PurposeData collectionNot an AI product
- TokenNo robots.txt tokenUse it in a User-agent line
robots.txt
The operator does not say whether this bot follows robots.txt.
It has no robots.txt token of its own. Control it at the server, firewall or CDN.
Quick facts
- User agent contains
- python-requests, python-urllib, aiohttp, httpx, Scrapy, curl/, Wget, Go-http-client, okhttp, axios, node-fetch, undici, Java/, Apache-HttpClient, libwww-perl, HeadlessChrome, PhantomJS, Puppeteer, Playwright
- Operator
- Unknown
- Category
- Other agents
- Purpose
- Data collection
- robots.txt
- Not documented
- Verification
- Cannot be verified
If you block it
Keeps your pages out of what this operator collects.
Check a visit is real
- No published way to check it. A request using this name is only a claim.
Where this comes from
The same registry NdovuSEOLabs uses to classify crawler traffic in your server and CDN logs.
Registry version 2026-10-29.1
See which of these crawlers visit your site.
Crawl Analytics reads your own server and CDN logs and checks each claim against the operator’s published IP ranges and DNS.