What is robots.txt?

robots.txt: definition and meaning

The robots.txt file is a simple text file in the root directory of a domain, for example at example.com/robots.txt. It is based on the Robots Exclusion Protocol, which was standardized in 2022 as RFC 9309. Reputable crawlers fetch the file before visiting a website and follow the rules it contains.

Every subdomain needs its own robots.txt, and the file is publicly readable: anyone can open it and see which paths are excluded. That makes it unsuitable for hiding confidential areas. Access protection, logins, and permission management are responsible for that.

How robots.txt works

The file consists of blocks. With User-agent you define which crawler a rule applies to, with Disallow you block paths, and with Allow you permit exceptions. Google evaluates only the first 500 KiB of the file and recognizes only these directives plus the Sitemap line. Instructions such as crawl-delay are ignored.

The difference between crawling and indexing is important. A page blocked with Disallow can still appear in search results if other pages link to it, though without a description. If a page should be removed from the index, it needs a noindex and must not be blocked, otherwise the crawler never sees the noindex.

• User-agent: * – applies to all crawlers
• Disallow: /cart/ – blocks a path from crawling
• Allow: – permits an exception inside a blocked path
• Sitemap: – gives the full URL of the XML sitemap

Real-world example: a relaunch with consequences

A skilled-trades business has its website rebuilt. On the test server, a Disallow blocks the entire site so nobody finds the unfinished version. At launch the file is accidentally copied over. A few weeks later, inquiries from Google drop noticeably because crawlers are no longer allowed to fetch anything.

The fix is a corrected file and a check in Google Search Console; after that it takes a while until Google fetches the pages in full again. That is why the robots.txt belongs on every go-live checklist. Test environments should additionally be protected with a password, not just with Disallow.

Limits and common mistakes

The robots.txt file is a request, not a security measure: reputable crawlers follow it, malicious bots simply ignore it. It is also error-prone, because a single misplaced line can lock out entire sections without the problem being obvious on the website itself.

• Disallow: / on the live site blocks the entire website
• Blocking CSS and JavaScript files: Google cannot render the page correctly
• Trying to “remove” pages via robots.txt: noindex is responsible for that
• Listing confidential paths: the file reveals them to every reader

AI crawlers and imple­menta­tion

Many site owners now also use robots.txt to allow or block AI crawlers such as GPTBot, ClaudeBot, or PerplexityBot. The Google-Extended token controls use for Google's Gemini models; regular Google Search is unaffected. Nano Concepts reviews the robots.txt file during technical SEO audits and aligns it with your goals.