Free robots.txt / sitemap.xml Validator
One of the free tools from KinetixSEO, an SEO and AI-citation (GEO) checker. Check any site's robots.txt for malformed directives, duplicate user-agent blocks, and AI-bot crawl rules, plus its sitemap.xml for well-formedness and lastmod coverage. These two files are the first thing every crawler — search engine or AI — reads before touching the rest of the site, so a mistake here can silently cap your visibility no matter how good the pages underneath are.
What you'll get
Results for
https://example.com
robots.txt
- Found
- Yes
- Blocks Googlebot
- No
- Disallow rules
- 3
Malformed lines
- Line 14: Disallow /admin
Sitemaps declared
- https://example.com/sitemap.xml
sitemap.xml
- Found
- Yes
- Child sitemaps
- 12
- Pages across them
- 48,213
- Has lastmod
- Yes
This is a sitemap index: 12 child sitemaps listing 48,213 pages between the shards we reached. We followed it 1 level deep and fetched 12 child sitemaps.
2 invalid lastmod value(s) found.
Child sitemaps
1 child sitemap is broken, listed first.
| Sitemap | Status | Well-formed | Pages | lastmod coverage |
|---|---|---|---|---|
https://example.com/sitemap-products-7.xml Level 1 | HTTP error (404) | No | — | — |
https://example.com/sitemap-posts-1.xml Level 1 | OK (200) | Yes | 4,000 | 99% |
https://example.com/sitemap-posts-2.xml Level 1 | OK (200) | Yes | 3,812 | 32% |
What the Free robots.txt / sitemap.xml Validator checks
- robots.txt exists, returns 200, and is served as plain text
- Directive syntax is valid — no malformed User-agent/Disallow/Allow lines
- Whether major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and others) are explicitly allowed, blocked, or unmentioned
- A Sitemap: directive points crawlers at the sitemap
- sitemap.xml exists, returns 200, and is well-formed XML against the sitemap protocol
- Sitemap URL count against the 50,000 URL / 50MB uncompressed limit
- Follows a sitemap index down into its child sitemaps (and their children), validating each shard individually rather than only the top-level file
- Reports which specific child sitemaps are unreachable, malformed, or oversized
- What share of sitemap URLs carry a <lastmod> date
- Sitemap URLs use consistent, canonical (non-parameterized) forms
Frequently asked questions
Does this tool check whether AI crawlers like GPTBot are blocked?
Yes — it reads your robots.txt and reports explicitly on the major AI crawlers by name: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, and others, distinguishing whether each is explicitly allowed, explicitly blocked with a Disallow rule, or simply unmentioned (which most crawlers treat as allowed by default). This matters separately from the classic Googlebot check most robots.txt tools focus on — a site can be fully open to Google and still have an AI-crawler block left over from a boilerplate robots.txt template or a security-minded default that predates AI search becoming a real traffic source.
What counts as a "malformed" robots.txt line?
The robots.txt spec expects each directive on its own line in a strict field: value format — User-agent: *, Disallow: /admin, and so on, each with the colon separator and a single value. This tool flags lines that break that shape: a missing colon, an unrecognized directive name, or a Disallow value that isn't a valid path. Most crawlers are lenient and will simply ignore a malformed line rather than erroring out, which is exactly why these mistakes go unnoticed for years — the file loads fine, nothing crashes, and the broken rule just silently does nothing.
Why does the tool check lastmod coverage in my sitemap.xml?
The <lastmod> tag tells crawlers when a URL last changed, which search engines use as one signal (among many) to decide how often to recrawl a page — a sitemap with accurate, up-to-date lastmod values can help freshly changed content get noticed faster. This tool reports what share of your sitemap URLs carry a lastmod value, and separately flags entries with an invalid lastmod format (not ISO 8601), since a malformed date is often worse than no date at all — some crawlers may simply disregard it as a broken signal.
What's the 50,000 URL / 50MB sitemap limit, and why does it matter?
The sitemap protocol caps a single sitemap file at 50,000 URLs or 50MB uncompressed, whichever comes first — a sitemap that exceeds either limit is invalid per spec, and crawler behavior when they encounter an oversized sitemap is inconsistent (some truncate, some skip it entirely). Large sites that hit this limit need to split into multiple sitemap files referenced from a sitemap index file rather than one giant sitemap.xml. This tool reports your current URL count against that ceiling so you can see how much headroom you have before it becomes a problem.
My robots.txt exists and my sitemap.xml exists — is that enough?
Existing and being valid are two different things, which is exactly the gap this tool is built to catch. A robots.txt can exist, return a 200, and still contain a directive that accidentally blocks Googlebot from your whole site, or a duplicate User-agent block that creates ambiguous rules. A sitemap.xml can exist and still be missing the <loc> tag on some entries, or reference URLs that don't match your site's actual canonical URL format. Both files need to be checked for correctness, not just presence — this tool checks both.
Related reading
Go deeper
This free check looks at one signal in isolation. KinetixSEO's paid report combines every signal below into one prioritised fix list and tracks it over time.
See the full paid report for this →This is one check from KinetixSEO's full SEO/GEO audit. Want the complete picture — Core Web Vitals, AI-citation readiness, technical SEO health, and tailored fixes? Run the free full checker.