robots.txt Validator

robots.txt controls which paths of a site search bots are allowed to crawl. A mistake in this file can accidentally block an entire site from indexing, or expose internal sections that shouldn't be crawled. This tool checks the file's structure and tests a specific path against allow/disallow rules.

How to use it

• Paste the contents of robots.txt and the tool parses the User-agent groups and directives (Allow, Disallow, Sitemap).
• Enter a specific path (e.g. /admin/ or /blog/post-1) to check whether it's allowed for a chosen bot.
• Syntax errors and suspicious constructs are flagged separately from valid directives.

Common uses

• Checking before a deploy that a new robots.txt hasn't accidentally blocked important pages.
• Debugging why a specific page isn't indexed — it may be caught by a Disallow rule.
• Comparing rules for different bots (Googlebot, Bingbot, etc.) when separate User-agent groups target them.

Things to keep in mind

robots.txt only blocks crawling — it doesn't guarantee a page won't appear in the index: if something else links to it, Google can still show the URL with no description. To reliably exclude a page, use a meta noindex tag, and the page must not be blocked in robots.txt for that tag to be seen.

Paths in robots.txt are case-sensitive — /Page and /page are treated as different paths.