robots.txt looks like a trivial text file, but it went almost three decades without an official standard. The "Robots Exclusion Protocol" was proposed informally in 1994 and only became a real IETF standard — RFC 9309 — in September 2022, once Google pushed to formalize what browsers and crawlers had been treating as a loose convention for years.
User-agent, Disallow and Allow
The file is made of blocks, each starting with User-agent:, followed by Disallow and Allow lines that mark paths as off-limits or explicitly permitted. A crawler applies the first block whose User-agent matches its own name, falling back to a User-agent: * block if no specific match exists.
What RFC 9309 actually standardized
Before 2022, different crawlers disagreed on edge cases: how long a line can be, what happens with conflicting rules, whether unsupported directives should be ignored or cause an error. RFC 9309 settled these details — for example, it caps parsed files at 500 kibibytes and specifies that unrecognized lines must simply be skipped rather than breaking the whole file.
Sitemap: a hint, not a rule
The Sitemap: directive points crawlers to a sitemap.xml file. It's optional, but convenient — a bot doesn't have to guess the sitemap's location or rely solely on it being registered in a search console.
Why this matters in practice
- Check robots.txt syntax before publishing, so a stray typo doesn't accidentally block an entire site from crawling.
- Confirm that important sections aren't being hidden by conflicting rules from different User-agent blocks.
- Test a specific user-agent and path combination instantly, instead of waiting for the next crawl to find out.
Why Disallow doesn't hide a page from search results
This is the most common confusion: Disallow stops a bot from crawling a page's content, but it doesn't stop that URL from being indexed. If another indexed site links to a blocked page, Google can still show its URL in results — without a description, since the content itself was never fetched.