robots.txt
Also known as: Robots Exclusion Protocol
In short
A website file that communicates path-based crawling rules to automated clients that follow the Robots Exclusion Protocol.
A robots.txt file communicates crawling rules for a website. It identifies user agents and paths they may or may not fetch under the Robots Exclusion Protocol.
The file manages cooperative crawler access. It is public information and does not authenticate visitors, protect confidential files, or enforce a security boundary.
A practical example
Imagine a retailer with public product pages and many filtered category URLs. It reviews whether crawlers need access to every filter combination while keeping important products accessible.
Before applying a broad rule, the team checks which paths match and whether required page resources are affected. This illustrative review prevents a rule intended for repetitive filters from also blocking useful product pages.
Crawling and indexing are different
Blocking a URL in robots.txt does not guarantee its removal from Google Search. Google can discover a blocked URL through links without crawling the page’s content.
An indexing instruction such as noindex serves a different purpose and needs to be accessible to the crawler that reads it. Private material belongs behind proper access controls.
For AI services, inspect the provider’s named bots and stated purposes. Search crawling, potential training collection, and user-triggered retrieval can have different controls. Test changes deliberately rather than assuming one generic rule produces the same outcome everywhere.