What is an AI crawler?
An AI crawler is a piece of automated software run by an AI company that visits websites and reads their pages, either to gather training material for models or to fetch content for answers in AI search and chat. Site owners control it through robots.txt.
Last updated
In more detail
Whether an AI assistant can mention your business accurately depends partly on whether its crawlers are allowed to read your site. Each company runs its own bots under published names, and site owners control them through robots.txt. OpenAI, for example, documents separate agents: GPTBot gathers content that may be used to train its models, while OAI-SearchBot is used to surface sites in ChatGPT's search features, and each can be allowed or disallowed on its own. That separation matters, because a site can decline training use and still appear in search answers. Review your robots.txt and any firewall or bot-protection settings to see what is currently permitted. The common mistake is a blanket block added years ago, or by a security tool's default, that quietly shuts out every AI crawler. The opposite mistake is assuming a page is readable when its text only appears after scripts run; a crawler that does not run scripts sees an empty page.
Source
The facts that can change were checked against developers.openai.com/api/docs/bots (opens in a new tab) in November 2026.
Related terms
- Robots.txtRobots.txt is a plain text file at the root of a website that tells search engine crawlers and other bots which parts of the site they may or may not request.→
- llms.txtllms.txt is a proposed convention for a text file on a website that gives AI models a short, curated guide to the site: what it is about and links to its most useful pages, written in Markdown.→
- AI citationAn AI citation is a reference an AI assistant or AI search feature attaches to its answer, naming or linking the web page that a statement came from.→
- Retrieval-augmented generation (RAG)Retrieval-augmented generation (RAG) is a method in which an AI system first searches a set of documents or the web for relevant passages, then writes its answer using what it found.→
More in AI search
Do it with Wonib
Put the words into practice
Wonib learns your brand once, then makes the posts, pictures, copy, decks and documents in it, checked against the brand before you see them. Start free with 1,000 credits, no card needed.
Last updated