SEO
robots.txt Validator
Check robots.txt — User-agent group structure, directives, and a path tester for allow/block decisions.
If you enter just a domain, /robots.txt is appended. This won't work for every site — some block cross-origin requests (CORS); if fetching fails, open robots.txt in a new tab and paste the contents manually.
robots.txt controls which paths of a site search bots are allowed to crawl. A mistake in this file can accidentally block an entire site from indexing, or expose internal sections that shouldn't be crawled. This tool checks the file's structure and tests a specific path against allow/disallow rules.
How to use it
- Paste the contents of robots.txt and the tool parses the User-agent groups and directives (Allow, Disallow, Sitemap).
- Enter a specific path (e.g. /admin/ or /blog/post-1) to check whether it's allowed for a chosen bot.
- Syntax errors and suspicious constructs are flagged separately from valid directives.
Common uses
- Checking before a deploy that a new robots.txt hasn't accidentally blocked important pages.
- Debugging why a specific page isn't indexed — it may be caught by a Disallow rule.
- Comparing rules for different bots (Googlebot, Bingbot, etc.) when separate User-agent groups target them.
Things to keep in mind
robots.txt only blocks crawling — it doesn't guarantee a page won't appear in the index: if something else links to it, Google can still show the URL with no description. To reliably exclude a page, use a meta noindex tag, and the page must not be blocked in robots.txt for that tag to be seen.
Paths in robots.txt are case-sensitive — /Page and /page are treated as different paths.
Article about this tool: robots.txt: how search bots decide what they can crawl
Frequently asked questions
Why is my Disallow rule not blocking the page I expected?
Robots.txt rules are matched by prefix, not by exact page, and more specific rules override general ones — a Disallow: /blog won't block /blog-archive unless you account for how prefix matching actually works, and an Allow rule elsewhere can override a broader Disallow.
Does robots.txt actually prevent a page from being indexed?
Not reliably by itself. It only asks well-behaved crawlers not to fetch a page — a disallowed page can still be indexed (without its content) if other pages link to it, so use a noindex meta tag or header for a real guarantee.
Is my robots.txt content sent anywhere to validate it?
No. Validation happens entirely in your browser — nothing is uploaded to a server.
Does Googlebot support wildcard characters * and $ in robots.txt?
Yes — Google supports * as any sequence of characters and $ as the end of a URL, though this is an extension beyond the original 1994 spec, so not every bot is guaranteed to recognize them.
Are paths in robots.txt case-sensitive?
Yes — /Admin/ and /admin/ are treated as different paths, so a Disallow rule needs to match the exact character case used in the site's real URLs.