EN
Start free
Start free

What is robots.txt?

Robots.txt is a plain text file at the root of a website that tells search engine crawlers and other bots which parts of the site they may or may not request. Well-behaved crawlers read it before visiting. Anyone can view the file.

Last updated

In more detail

The file sets the rules of entry for automated visitors. Google describes its purpose as telling crawlers which URLs they can access, mainly to avoid overloading a site with requests. Each rule names a crawler, or all crawlers, and lists paths to allow or disallow. Site owners use it to keep bots out of search-result pages, shopping baskets and admin areas, and increasingly to choose which AI crawlers may fetch their content. The file is public: anyone can read it by adding /robots.txt to a site's address. The common mistake is using it to hide a page from search results. Google states that robots.txt is not a mechanism for keeping a web page out of Google; a blocked page can still be listed if other sites link to it. Use a noindex tag or a password for that. The costliest mistake is one stray line, "Disallow: /", which blocks the whole site.

Source

The facts that can change were checked against developers.google.com/search/docs/crawling-indexing/robots/… (opens in a new tab) in November 2026.

Put the words into practice

Wonib learns your brand once, then makes the posts, pictures, copy, decks and documents in it, checked against the brand before you see them. Start free with 1,000 credits, no card needed.

Last updated

Start free with 1,000 credits

No card needed. Sign up with Google or your email, confirm it, and make your first piece in minutes.

Continue with Google

or

Already have an account? Sign in