Other meanings of Robots exclusion standard
WEB STANDARDS
The Robots exclusion standard is a protocol for web crawlers to avoid indexing certain pages by publishing instructions in a site’s /robots.txt file. It provides voluntary access guidance rather than a security barrier: compliant crawlers normally follow the rules, while malicious or poorly implemented software can ignore them.
The standard gives website operators a machine-readable way to state which automated agents may request particular URL paths. The file is normally placed at the top level of an origin, such as https://example.com/robots.txt, and consists of records containing a User-agent field followed by directives such as Disallow and, in some implementations, Allow.1
A record can target all crawlers with User-agent: * or identify a particular crawler. Path rules are evaluated against URLs on that origin; they do not grant permissions to other sites, conceal content from people, or protect files from direct requests. A blank Disallow value conventionally permits crawling, while Disallow: / requests that the entire site not be crawled.
The protocol governs crawling more directly than indexing. A URL mentioned elsewhere may still appear in a search index even when its contents were not fetched, so private or removal-sensitive material requires authentication, access controls, or an appropriate noindex mechanism.2
The Robots Exclusion Protocol began as an informal convention in 1994, when web administrators and crawler authors sought a simple way to reduce unwanted automated access. Its early development was collaborative rather than the product of a formal standards body, and implementations accumulated extensions over time.
RFC 9309, published by the Internet Engineering Task Force in 2022, documented and standardized the long-established protocol’s core behavior under the name Robots Exclusion Protocol.1 The RFC specifies the file’s location, parsing model, groups, matching rules, and treatment of common retrieval outcomes. It also recognizes that crawlers may apply additional policies outside the protocol, including rate limits and search-engine-specific directives.
The standard is therefore best understood as a compatibility specification, not a complete crawler-management language. Search services may document supported extensions, including sitemap declarations, but such extensions should not be assumed to work universally.3
Compliant crawlers fetch and interpret /robots.txt before requesting URLs covered by its rules, usually caching the result to avoid repeatedly loading the file. Matching is based on URL paths, and the standardized syntax permits patterns involving end-of-path markers and wildcards; unsupported or malformed conventions can produce different results across crawlers.1
Robots rules are scoped to an origin and can vary by scheme, host, and port. A rule on www.example.com does not automatically govern api.example.com. Redirects, server errors, network failures, and unreachable files can also affect crawler decisions, so operators should monitor the file as part of site operations rather than treating it as static decoration.
Google describes robots.txt as controlling crawling, not as a reliable method for keeping a page out of search results.2 Search engines may also impose independent policies concerning duplicate URLs, crawl budget, spam, or legal removal requests.
The most consequential limitation is that robots.txt is public and advisory, not confidential. Anyone can retrieve it, and a hostile crawler can disregard it; sensitive resources should instead use authentication, authorization, network restrictions, or encryption.
The file can unintentionally disclose site structure by naming administrative, staging, or legacy paths. Blocking a directory does not prevent discovery of its URL through links, referrers, browser history, or other datasets. Conversely, disallowing a URL can make it harder for a search engine to see a page-level noindex directive, because the crawler may be prevented from fetching the page where that directive appears.2
Another overlooked detail is that robots.txt is not a universal rate-limiting protocol. Operators needing gentler traffic can combine crawler-specific documentation, server-side throttling, HTTP response controls, and contact channels. Sitemaps may be declared in robots.txt, but they describe URLs for discovery; they do not override crawl restrictions or guarantee indexing.3
Robots.txt directives express crawler preferences and operational guidance; they should never be used as the sole protection for confidential or security-sensitive information.
Help improve the encyclopedia. Reports go straight to the site manager.