CONNECT · WORDPRESS CRAWL-LOG PLUGIN
Your own WordPress logs, connected in minutes.
The NdovuSEOLabs Crawl Log plugin sends verified crawler activity, optional site-health requests, a WooCommerce catalogue sync and consented traffic signals from a WordPress site to its project. It works on ordinary hosting, with or without WP-Cron.
- Signed batches
- Query strings dropped
- No prices, customers or orders
- OnCrawler requestsSigned batches every few minutes
- OnSite HealthOrdinary requests counted, not stored
- OnCatalogue syncDaily, in parts of 500 products
- OffTraffic signalsTurn on after consent is set up
Crawlers
Googlebot totals and automated requests
Site health
status and response times by route
Catalogue
daily WooCommerce product sync
Consent
traffic signals only after consent
INSIDE THE PRODUCT
Everything Crawl Analytics needs, from one plugin.
Install it, connect it with a project token, and your logs start arriving.
Crawlers
Send crawler evidence from WordPress
Googlebot daily totals plus a small record for each automated request, posted in signed batches and verified in the app. Query-string values are dropped before anything is stored.
- Googlebot daily totals
- Automated requests, verified in the app
- Query-string values dropped
| Batch | Events | Status |
|---|---|---|
| 10:05 | 412 | Accepted |
| 10:00 | 388 | Accepted |
| 09:55 | 0 | Heartbeat |
| 09:50 | 401 | Accepted |
Site health & catalogue
Add site health and your product catalogue
Optionally send ordinary requests for status codes and response times by route. Stores sync product ids, URLs, SKUs, stock and categories daily, with no prices, customers or orders. Plugin, theme and core updates arrive as site events.
- Status and response time by route
- Daily WooCommerce catalogue sync
- Updates sent as site events
Reliable sending
Built for ordinary hosting
It sends through WP-Cron or Action Scheduler, or on its own when neither runs, with a heartbeat when nothing is queued so a quiet site is not mistaken for a broken connection.
- WP-Cron or Action Scheduler
- Sends without either
- Heartbeat when idle
Lightweight
Built to stay out of the visitor’s way
The plugin finishes the response before it does any work, adds each record with one insert into a capped queue, and sends batches every 5, 15 or 60 minutes.
- Work happens after the response
- Queue capped at 20,000 records
- Batches every 5, 15 or 60 minutes
- OKLast batch3 minutes ago, 214 records
- OKHeartbeatSite reports normally
- OKGooglebot daily totalsSent for 27 Oct
- InfoIntervalEvery 15 minutes
Proxies & cli
Trustworthy addresses behind Cloudflare or a proxy
Behind Cloudflare, the visitor IP is only believed when the connection comes from Cloudflare’s published addresses. Other proxies must be listed as trusted. Settings can be locked in wp-config.php and tested with WP-CLI.
- Cloudflare and trusted-proxy support
- wp-config.php constants
- WP-CLI send, bots, catalog and status
| Setting | Value | Source |
|---|---|---|
| Visitor IP from | Cloudflare | Constant |
| Send ordinary requests | On | Settings |
| Catalogue sync | On | Settings |
| Interval | 15 minutes | Constant |
WHAT IT MEASURES
What the plugin sends, and what it never sends.
What each metric tells you, and where it comes from.
| Metric | What it tells you | Source |
|---|---|---|
| Googlebot daily totals | Requests per day, responses, file types, Googlebot types and most-crawled URLs, verified by DNS. | Your WordPress server |
| Automated request records | A small record for each request that looks automated, verified later in the app. | Your WordPress server |
| Site health (optional) | Status codes, response times and traffic by route; visitors are counted, not stored as events. | Your WordPress server |
| Catalogue sync (WooCommerce) | Product and variation ids, URLs, type, SKU, status, stock, visibility, category, brand and dates. | WooCommerce |
| Site events | Plugin, theme and core updates. | WordPress |
| Browser signals (optional) | A few categorical automation indicators per page view, after consent. | Visitor’s browser |
| Never sent | Query-string values, prices, customers, orders, form values, keystrokes or fingerprints. | By design |
UNDER THE HOOD
How the plugin collects and sends.
How the numbers are produced, so you can judge them for yourself.
Googlebot checked on your server
Googlebot requests are verified by reverse and forward DNS on your server, cached per address and capped at 300 new lookups an hour.
Other bots verified in the app
Other automated requests are recorded as small queue records and verified by Ndovu against operators’ IP ranges or DNS.
Signed batches with a heartbeat
Batches are signed with the project token and sent every 5, 15 or 60 minutes through WP-Cron or Action Scheduler, or on their own when neither runs. A heartbeat tells a quiet site from a broken connection.
Catalogue in parts
On WooCommerce, the daily catalogue sync sends products and variations in parts of 500, without prices, customers or orders.
HOW IT WORKS
Connected in minutes, verified the same day.
Four steps from connecting your data to a decision you can stand behind.
Download the plugin
Download it from the project’s data sources and upload it in WordPress.
Connect with a token
Mint a project token; the connect link fills the endpoint and token for you to review.
Choose the options
Turn on site health, the catalogue sync or consented browser signals if you need them.
Test and confirm
Send the last 7 days and queued events now, then check real events arrive.
WITH YOUR AI AGENT
Ask whether the plugin is delivering.
Your agent reads the data the plugin sends, as Crawl Analytics and Traffic Quality evidence.
- Uses the same data as the web app
- Paid tools estimate their cost first
- Saves the report to the project
YouIs the WordPress plugin on yourstore.co.ke sending everything it should?
get_request_intelligenceget_bot_activity
AgentYes. Batches arrive every 15 minutes, the heartbeat is current and the catalogue sync ran this morning with 640 products. Site health is off, so response times by route are not available.
YouWhat would turning on site health add?
AgentStatus codes and response times by route for all visitors, so server errors on checkout or category pages show up even when no crawler hits them.
Illustration · sample data
USE CASES
Built for the way you work.
How teams put it to work.
See crawler attention on products
Combine the catalogue sync with crawler logs to see which products crawlers and AI agents reach.
Connect client sites quickly
Install, paste the endpoint and connect with a project token from the app.
See which products crawlers miss
The catalogue sync lets Crawl Analytics compare every product and variation with what Googlebot requested.
Deploy it like infrastructure
Lock settings with wp-config.php constants, install it as a must-use plugin and check it with WP-CLI.
DATA & LIMITS
Know what the numbers can and cannot tell you.
Where the data comes from, how fresh it is, and the questions it cannot answer on its own.
| Source | What it provides | Freshness & limits |
|---|---|---|
| WordPress | Requests that reach WordPress, site events. | Origin only: cannot see CDN cache hits or edge blocks. |
| WooCommerce | Catalogue sync in parts of 500 products. | Daily; no prices, customers or orders. |
| Visitor browsers (optional) | Consented automation signals. | Never on cart, checkout, account, login or admin pages. |
Origin only
Requests a CDN serves from cache, or blocks at the edge, never reach WordPress. Add a Cloudflare source to see them.
Missing is not quiet
A gap can mean a collection problem; the heartbeat and connections view tell the two apart.
One token at a time
Replacing a token revokes the previous one.
Queue limits
If the queue fills before sending, new records are dropped rather than slowing the site.
WORKS WITH
Connected to the tools you already use.
The sources and clients this feature reads from or works alongside.
CHOOSE THE RIGHT VIEW
Which view answers which question.
Start where the question points. Each view keeps its own source.
| Your question | Start with | Why |
|---|---|---|
| Which crawlers visit my WordPress site? | Crawl Analytics | Verified crawler evidence. |
| Is my traffic real? | Traffic Quality | Browser signals and GA4. |
| Do I need edge data too? | Data sources | Compare what each source can see. |
LEARN & TRY
Guides and free tools for the same job.
Read how the evidence works, or try a free check before you connect a project.
Crawler directory
Every crawler, its operator, its robots.txt token and how to verify it.
AI crawler log analyzer
Paste log lines and count crawler requests.
FAQ
WordPress plugin: common questions.
Straight answers about what it does, what it costs and where its limits are.
How do I connect a site?
Download the plugin from the project’s data sources, install it, then mint a token and choose “Open WordPress and connect”. The settings fill in for you.
What happens if I replace the token?
Replacing a token revokes the previous one, so access changes are deliberate.
Does the plugin slow the site down?
Each request adds one small database write after the response is sent. Busy sites can shorten the send interval.
Does it collect personal data?
Query-string values are dropped, ordinary visitors are only counted, and the catalogue sync sends no prices, customers or orders. Traffic signals need consent and hold no identifiers.
Does the plugin slow my site?
It is built not to: work happens after the response is sent, each record is one insert, and DNS checks are cached and capped.
Does it work behind Cloudflare?
Yes. Set “Visitor IP from” to Cloudflare; the header is trusted only from Cloudflare’s published addresses.
Can I use WP-CLI?
Yes: wp openseo-crawl send, bots, catalog and status.
Does the catalogue sync send prices?
No. It sends product ids, URLs, SKU, status, stock, visibility, category, brand and dates, never prices, customers or orders.
Can I run it as a must-use plugin?
Yes. It works from wp-content/mu-plugins/, with settings locked by constants in wp-config.php.
What are the limits of the data?
Replacing a token revokes the previous one. Missing events can mean a collection problem rather than no crawling; the connections view shows what each source can see.
How do I get access?
Platform access is reviewed. Request access with your website and the data sources you use, and the team will help you connect them.
See it with your own data.
Tell us about your website. We will review access and help you connect the right sources.