SITEMAPS & CRAWLING
Mini Website Crawler
A crawler sees a site the way a search engine does: it starts at one page, follows the links and records what each page answers. This one crawls a small sample, enough to spot site-wide problems such as a template with no H1, a menu full of redirects or a section left on noindex.
- Checks the live page
- Public addresses only
- Free, no sign-up
- CheckStatus codes and redirects of every crawled page
- CheckMissing and duplicate titles, missing H1
- Checknoindex and canonicals pointing elsewhere
- CheckPages with under 100 words
Free
no sign-up
Live
fetched when you run the check
5 checks
what the tool looks at
30 a minute
checks per visitor, to keep it free
Loading the tool…
HOW IT WORKS
How to use the Mini Website Crawler
One public URL to start from.
Enter the homepage, or the page whose section you want to check.
Read the summary of problems, then scan the table.
Download the CSV to sort and filter it in a spreadsheet.
WHAT IT CHECKS
What it looks at, and how to read it.
A summary of errors, redirects and missing basics, and a table of every crawled page you can download as CSV.
What it checks
- Status codes and redirects of every crawled page
- Missing and duplicate titles, missing H1
- noindex and canonicals pointing elsewhere
- Pages with under 100 words
- robots.txt rules for our crawler, and an unreachable robots.txt
How to read the results
- Problems repeated on many pages usually come from one template, so one fix clears them all.
- A redirect in the table means a page links to an old address; the link still works, but crawlers take an extra hop.
- noindex and canonicals are reported, not judged: many are deliberate.
The concepts behind it, in the SEO glossary: Web crawler.
GOOD TO KNOW
Common mistakes, an example and the limits.
Common mistakes
- Taking a 50-page sample as a full audit. Use it to find patterns, not to count every issue.
- Starting from a page deep in the site and expecting the whole site to be covered.
A worked example
A crawl of a shop finds 14 product pages with the same title, all from a category template that prints the category name instead of the product name.
Limitations
- Crawls at most 50 pages, breadth first from the start URL, so on a bigger site it covers the pages closest to it.
- Follows links in the HTML the server sends. Links a page adds with JavaScript, and pages linked only from forms or scripts, are not found.
- Stays on the start URL’s host and obeys its robots.txt for our crawler; blocked pages are counted, not fetched.
FAQ
Questions about the Mini Website Crawler
Straight answers about what it does and where its limits are.
Which user agent does it crawl as?
NdovuSEOLabs-FreeTools. robots.txt rules for that name, or for *, are obeyed.
Can it crawl more than 50 pages?
Not here. The Ndovu platform’s site audit crawls whole sites on a schedule.
Crawl your whole site on a schedule.
Free tools check one thing at a time. The NdovuSEOLabs platform connects your own Search Console, analytics and server logs, and keeps checking every page.