AI crawler
Also known as: AI bot, Training crawler, Search crawler
In short
An automated client that retrieves web content for an AI-related service, such as search indexing or potential model training.
An AI crawler retrieves web pages for a service that uses AI. The label covers different purposes, so identify the specific bot before deciding what its visit means.
For example, OpenAI documents separate bots for search and potential training use. Anthropic also distinguishes its training crawler, search bot, and user-directed retrieval. A request made for a user is not necessarily automatic crawling.
A practical example
Imagine a retailer finding repeated bot requests in its server logs. The team verifies the requesting service and checks which product URLs were fetched, along with the returned status codes.
A successful request shows that content was delivered to that client. It does not establish that the page was indexed, used for training, cited in an answer, or seen by a shopper. Those are separate outcomes in this illustrative investigation.
Apply controls by purpose
Review each provider’s documentation before changing access rules. A decision about model training does not necessarily require the same decision about search discovery.
Do not assume every automated request follows identical robots.txt behavior. Providers distinguish crawlers, fetchers, and user-operated browser agents. Keep your access policy intentional, verify the effect in logs, and avoid reporting bot visits as human website traffic or evidence of commercial demand.