EN
Start free
Start free

What is an AI crawler?

An AI crawler is a piece of automated software run by an AI company that visits websites and reads their pages, either to gather training material for models or to fetch content for answers in AI search and chat. Site owners control it through robots.txt.

Last updated

In more detail

Whether an AI assistant can mention your business accurately depends partly on whether its crawlers are allowed to read your site. Each company runs its own bots under published names, and site owners control them through robots.txt. OpenAI, for example, documents separate agents: GPTBot gathers content that may be used to train its models, while OAI-SearchBot is used to surface sites in ChatGPT's search features, and each can be allowed or disallowed on its own. That separation matters, because a site can decline training use and still appear in search answers. Review your robots.txt and any firewall or bot-protection settings to see what is currently permitted. The common mistake is a blanket block added years ago, or by a security tool's default, that quietly shuts out every AI crawler. The opposite mistake is assuming a page is readable when its text only appears after scripts run; a crawler that does not run scripts sees an empty page.

Source

The facts that can change were checked against developers.openai.com/api/docs/bots (opens in a new tab) in November 2026.

Put the words into practice

Wonib learns your brand once, then makes the posts, pictures, copy, decks and documents in it, checked against the brand before you see them. Start free with 1,000 credits, no card needed.

Last updated

Start free with 1,000 credits

No card needed. Sign up with Google or your email, confirm it, and make your first piece in minutes.

Continue with Google

or

Already have an account? Sign in