SITEMAPS & CRAWLING
Sitemap vs Crawl Comparison
A sitemap should list exactly the pages you want indexed, at their final URLs. In practice sitemaps drift: they keep removed products, list redirected addresses, or miss a whole section a plugin does not know about. This tool reads the sitemaps and crawls the site, then compares the two.
- Checks the live page
- Public addresses only
- Free, no sign-up
- CheckSitemap URLs that redirect or answer an error
- CheckSitemap URLs with noindex or a canonical to another URL
- CheckIndexable crawled pages missing from the sitemap
- CheckSitemap URLs the crawl did not reach
Free
no sign-up
Live
fetched when you run the check
4 checks
what the tool looks at
30 a minute
checks per visitor, to keep it free
Loading the tool…
HOW IT WORKS
How to use the Sitemap vs Crawl Comparison
One public URL on the site.
Enter the site’s homepage.
Fix sitemap URLs reported as redirects, errors, noindex or canonicalised.
Check why the sitemap leaves out the indexable pages listed.
WHAT IT CHECKS
What it looks at, and how to read it.
Sitemap URLs that should not be there, indexable pages the sitemap leaves out, and counts from both sides.
What it checks
- Sitemap URLs that redirect or answer an error
- Sitemap URLs with noindex or a canonical to another URL
- Indexable crawled pages missing from the sitemap
- Sitemap URLs the crawl did not reach
How to read the results
- Bad sitemap URLs make search engines trust the sitemap less, so fix them first.
- A sitemap URL the crawl did not reach is not necessarily an orphan; the crawl stops at 50 pages.
The concepts behind it, in the SEO glossary: Indexing, XML sitemap.
GOOD TO KNOW
Common mistakes, an example and the limits.
Common mistakes
- Listing http:// or non-www URLs that redirect to the real ones.
- Leaving noindex pages, such as cart or account pages, in the sitemap.
A worked example
A shop’s sitemap lists 40 discontinued products that now redirect to their category, while its new blog section is missing because the SEO plugin was never told about it.
Limitations
- Crawls at most 50 pages, breadth first from the start URL, so on a bigger site it covers the pages closest to it.
- Follows links in the HTML the server sends. Links a page adds with JavaScript, and pages linked only from forms or scripts, are not found.
- Stays on the start URL’s host and obeys its robots.txt for our crawler; blocked pages are counted, not fetched.
- Reads up to 10 sitemap files and 500 URLs from each. When the sitemaps hold more, pages missing from them are not reported.
FAQ
Questions about the Sitemap vs Crawl Comparison
Straight answers about what it does and where its limits are.
Which sitemaps does it read?
Those named in robots.txt, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and /sitemap-index.xml, following one level of sitemap index.
Need this for a whole website, over time?
The NdovuSEOLabs platform connects your own Search Console, analytics and server logs, and audits every page on a schedule.