What is robots.txt?
Robots.txt is a plain text file at the root of a website that tells search engine crawlers and other bots which parts of the site they may or may not request. Well-behaved crawlers read it before visiting. Anyone can view the file.
Last updated
In more detail
The file sets the rules of entry for automated visitors. Google describes its purpose as telling crawlers which URLs they can access, mainly to avoid overloading a site with requests. Each rule names a crawler, or all crawlers, and lists paths to allow or disallow. Site owners use it to keep bots out of search-result pages, shopping baskets and admin areas, and increasingly to choose which AI crawlers may fetch their content. The file is public: anyone can read it by adding /robots.txt to a site's address. The common mistake is using it to hide a page from search results. Google states that robots.txt is not a mechanism for keeping a web page out of Google; a blocked page can still be listed if other sites link to it. Use a noindex tag or a password for that. The costliest mistake is one stray line, "Disallow: /", which blocks the whole site.
Source
The facts that can change were checked against developers.google.com/search/docs/crawling-indexing/robots/… (opens in a new tab) in November 2026.
Related terms
- XML sitemapAn XML sitemap is a file on a website that lists the pages its owner wants search engines to find.→
- AI crawlerAn AI crawler is a piece of automated software run by an AI company that visits websites and reads their pages, either to gather training material for models or to fetch content for answers in AI search and chat.→
- llms.txtllms.txt is a proposed convention for a text file on a website that gives AI models a short, curated guide to the site: what it is about and links to its most useful pages, written in Markdown.→
- Canonical tagA canonical tag is a line in a page's HTML that tells search engines which address is the main version of a page when the same or very similar content can be reached at more than one address.→
More in SEO
Do it with Wonib
Put the words into practice
Wonib learns your brand once, then makes the posts, pictures, copy, decks and documents in it, checked against the brand before you see them. Start free with 1,000 credits, no card needed.
Last updated