AI crawlers do not read your site the way Googlebot does — and most robots.txt files still have no idea GPTBot, ClaudeBot or PerplexityBot exist. This is a hands-on technical guide to configuring both files correctly, with Nevatrix's own live setup as a real example.
robots.txt controls which crawlers may access your site, while llms.txt is an optional summary file that points AI systems to your most useful pages. For AI visibility, robots.txt matters most: if it blocks crawlers such as OAI-SearchBot or PerplexityBot, those tools cannot read or cite your pages. This guide shows how to set up both.
Key takeaways
- robots.txt is a formal standard (RFC 9309) that well-behaved crawlers follow; llms.txt is a voluntary proposal that only some AI tools read.
- Each AI company runs separate crawlers for training, search and user-requested fetches, so you can allow some and block others.
- Blocking Google-Extended does not remove you from Google Search or AI Overviews; those use Googlebot.
- Most robots.txt files were written before AI crawlers existed, so their AI policy is accidental rather than chosen.
- Check your CDN or firewall too: many sites block AI crawlers there even when robots.txt allows them.
This guide explains what each file does, which AI crawlers exist in 2026, how to write both files, and the configuration running on nevatrix.com today. It is written for site owners and developers who want AI search tools to read their site deliberately rather than by accident.
What Is robots.txt and How Do AI Crawlers Use It?
robots.txt is a plain-text file at the root of your site that tells crawlers which paths they may or may not request. It uses User-agent lines to name a crawler and Allow or Disallow rules to set its access. The format is standardised as the Robots Exclusion Protocol, RFC 9309, and Google explains how it reads the file in its introduction to robots.txt.
AI crawlers read the same file, but each one has its own name. If your robots.txt has no rule for a crawler, the rules under "User-agent: *" apply to it. That is why an old robots.txt written for search engines can unintentionally allow every AI crawler, or block them all.
robots.txt is a request, not a lock. Reputable crawlers follow it, but it does not stop a determined scraper. If you need to keep content private, put it behind a login instead. Also remember that robots.txt is public: anyone can read it, so listing secret paths in it can draw attention to them rather than hide them.
What Is llms.txt and Does Your Site Need One?
llms.txt is a proposed Markdown file at yourdomain.com/llms.txt that gives AI systems a short, structured guide to your site: who you are, what you offer and which pages matter most. It is described at llmstxt.org and is voluntary; it is not an official web standard and no AI company is obliged to read it.
You do not strictly need one. Google says that no special files are required for its AI features and that normal SEO practices apply, as set out in its guidance on AI features and your website. An llms.txt file is still a cheap, low-risk addition. It takes an hour to write, helps AI tools and agents that do read it, and forces you to state your key facts clearly, which also improves your normal pages.
| robots.txt | llms.txt | |
|---|---|---|
| Purpose | Grants or denies crawler access | Summarises your site for AI systems |
| Status | Formal standard (RFC 9309) | Voluntary community proposal |
| Who reads it | All reputable crawlers | Some AI tools and agents |
| Effect if missing | All crawlers may access everything | No effect on access |
| Risk if wrong | Can block search engines or AI tools entirely | Can give AI tools outdated facts |
Which AI Crawlers Should You Know About in 2026?
The main AI crawlers come from OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, Meta and Common Crawl, and most companies run separate bots for training and for search or live fetches. The table lists the user-agent names their own documentation publishes.
| Crawler (user-agent) | Operator | What it is used for |
|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used to train models |
| OAI-SearchBot | OpenAI | Indexing pages for ChatGPT search results |
| ChatGPT-User | OpenAI | Fetching a page when a user asks ChatGPT to |
| ClaudeBot | Anthropic | Collecting content that may be used to train models |
| Claude-SearchBot | Anthropic | Indexing pages to improve search results in Claude |
| Claude-User | Anthropic | Fetching a page when a user asks Claude to |
| PerplexityBot | Perplexity | Indexing pages for Perplexity answers |
| Perplexity-User | Perplexity | Fetching pages in response to a user's question |
| Google-Extended | Controlling use of content for Gemini models; not used for Search ranking | |
| Applebot-Extended | Apple | Controlling use of content for Apple's AI models |
| CCBot | Common Crawl | Building a public web dataset used by many AI developers |
Check the official pages before relying on any list, because names change: OpenAI's crawler documentation, Anthropic's crawler policy, Perplexity's crawler guide, Google's list of common crawlers and Apple's Applebot page. Older names such as "anthropic-ai" and "Claude-Web" have been replaced by the Claude crawlers above.
The Accidental Policy Problem
The Accidental Policy Problem is our name for the situation on most websites: nobody ever decided whether AI tools may read the site. The robots.txt was generated years ago by a CMS or plugin, and its AI policy is whatever the "User-agent: *" group happens to say. Fixing it is simple: decide which crawlers you want, then name each one explicitly so your policy is a choice.
Should You Allow or Block AI Crawlers?
Most businesses that want customers to find them should allow AI search and user-fetch crawlers, then decide separately about training crawlers. Being readable by ChatGPT search, Claude and Perplexity gives you a chance to be cited in their answers; blocking them removes that chance.
| Your goal | Search and user-fetch crawlers | Training crawlers |
|---|---|---|
| Maximum AI visibility for a service business | Allow | Allow |
| Visibility in AI answers, but no training use | Allow (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) | Block (GPTBot, ClaudeBot, CCBot) |
| Protect paid or exclusive content | Block on paywalled paths | Block |
| Stay out of AI tools entirely | Block | Block, and accept lost visibility |
Two cautions apply. First, some providers say user-initiated fetchers may not follow robots.txt in the same way as automated crawlers, so read each provider's documentation. Second, blocking a training crawler does not remove content a model has already learned.
How to Configure robots.txt for AI Crawlers
You configure robots.txt for AI by adding an explicit group for each crawler you want to allow or block, below your general rules. These steps take about half an hour.
Step 1: Read your current file
Open yourdomain.com/robots.txt and note any "Disallow: /" lines, especially under "User-agent: *". A site-wide disallow there blocks every crawler without its own group.
Step 2: Decide your policy
Use the table above to choose which search, fetch and training crawlers you want, and write the decision down so it can be reviewed later.
Step 3: Add explicit groups
Add one group per crawler, or list several user-agents above one shared set of rules. A minimal group that allows AI search and fetch crawlers looks like this:
Example robots.txt group for AI search and fetch crawlers
- User-agent: OAI-SearchBot
- User-agent: ChatGPT-User
- User-agent: Claude-SearchBot
- User-agent: Claude-User
- User-agent: PerplexityBot
- User-agent: Perplexity-User
- Allow: /
- Disallow: /admin/
Step 4: Keep the sitemap line
End the file with a Sitemap line pointing to your XML sitemap, so every crawler can find your pages.
Step 5: Check your CDN and firewall
Bot-protection settings at your host or CDN can block AI crawlers even when robots.txt allows them. Check the security settings and logs for blocked requests from the crawlers you allowed.
Step 6: Test and review
Use Google Search Console's robots.txt report to confirm the file is valid, then review the policy every few months as new crawlers appear.
A copyable checklist version:
- 1Current robots.txt read and site-wide blocks noted.
- 2Policy for search, fetch and training crawlers written down.
- 3Explicit groups added for each chosen crawler.
- 4Sitemap line present at the end of the file.
- 5CDN and firewall bot settings checked.
- 6File validated and a review date set.
How to Write an llms.txt File
An llms.txt file is a short Markdown document with your name, a one-line summary, and grouped links to your most important pages, each with a brief description. Keep it factual: AI systems quote specifics such as services, locations and prices, not adjectives.
- First line: a level-one heading with your business name.
- Summary: one blockquote sentence saying what you do and for whom.
- Key facts: location, year founded, main services and starting prices that match your website exactly.
- Sections of links: for example Services, Guides and Company, each a list of links with one-line descriptions.
- Optional section: lower-priority links that AI tools can skip when they have limited space.
Example llms.txt structure for a service business
- # Your Business Name
- > One sentence on what you do, for whom and where.
- Key facts: city, year founded, main services, starting prices.
- ## Services: links to each service page with a one-line description.
- ## Guides: links to your most useful articles with one-line descriptions.
- ## Company: about, contact and pricing pages.
- ## Optional: lower-priority pages AI tools can skip.
Which pages should llms.txt link to?
Link to the pages you would want an AI tool to quote: service pages with prices, location pages, your most complete guides, your about page and your contact page. Leave out thin tag pages, old news posts and anything you would not want summarised. Twenty to forty well-described links is plenty for most business sites; a list of every URL defeats the purpose, which is to point AI tools at your best material first.
Update llms.txt whenever prices, services or key pages change. An out-of-date summary is worse than none, because AI tools may repeat its old facts. If you write about AI search more broadly, our guide to generative engine optimization explains how these files fit into a wider visibility plan.
Does robots.txt Affect Google AI Overviews?
Yes, but only through Googlebot: if Googlebot can crawl and index a page, it can appear in Google's AI Overviews, and if it cannot, it cannot. Google-Extended has no effect on Search or AI Overviews. Google also offers snippet controls, such as the nosnippet meta tag, that limit how much of a page can be shown, and these apply to AI features as well as normal results. If you want to understand how Google picks which pages to cite, our guide to getting cited in Google AI Overviews covers the content side.
How Do You Know If AI Tools Are Reading Your Site?
You can tell AI tools are reading your site from two places: crawler visits in your server or CDN logs, and visitors arriving from AI tools in your analytics. Checking both every month shows whether your configuration is working.
- Server and CDN logs: filter requests by user-agent for names such as OAI-SearchBot, ClaudeBot or PerplexityBot. Regular visits mean the crawler can reach you; blocked or error responses mean something is stopping it.
- Referral traffic: in your analytics, look for visits referred from chatgpt.com, perplexity.ai, claude.ai and gemini.google.com. These are people who clicked a link in an AI answer.
- Manual checks: ask the AI tools questions your customers would ask and note whether your business or pages are mentioned.
- Search Console: AI Overviews traffic is included in your normal Google Search performance data.
If crawlers visit but you never appear in answers, the problem is usually content rather than configuration. Pages need specific, quotable facts and a clear structure; our comparison of SEO, GEO and AEO explains what each discipline adds.
What Else Helps AI Tools Understand Your Site?
Beyond robots.txt and llms.txt, AI tools rely on the same signals as search engines: an up-to-date sitemap, structured data, fast server-rendered pages and clear, specific writing. These usually matter more than llms.txt itself.
- XML sitemap: lists every page you want found and when it last changed, so crawlers spend their time on current pages.
- Structured data: schema markup such as Organization, LocalBusiness, FAQPage and Article tells machines exactly what a page is about; Google's introduction to structured data explains the formats.
- Server-rendered content: many AI crawlers do not run JavaScript, so text that only appears after scripts load may be invisible to them.
- Consistent facts: the same name, address, prices and services on your website, Google Business Profile and directories make it easier for AI tools to trust what they read.
- Quotable passages: short paragraphs that answer one question each are easier for AI tools to lift into an answer than long, mixed sections.
If a crawler can reach your site but your pages are slow, script-heavy or vague, it will read them and still choose other sources. Configuration opens the door; content decides whether you are cited.
Real Example: The robots.txt and llms.txt on nevatrix.com
nevatrix.com allows major search engines and AI crawlers explicitly and publishes an llms.txt file. Our robots.txt has a general group, a group for search engines, and a separate group naming AI crawlers including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended and CCBot, each allowed with only private paths such as /api/ and /admin/ disallowed. A few aggressive scrapers that bring no visitors are blocked.
Our llms.txt lists our services, locations and main guides with one-line descriptions. We chose to allow training crawlers as well as search crawlers, because for a service business being understood by AI tools is worth more than keeping public marketing pages out of training data. A publisher selling content might sensibly decide the opposite.
When Allowing AI Crawlers Is Not Worth It
Blocking AI crawlers is the right call when your content is your product, or when crawling costs outweigh any visibility benefit.
- Paywalled or subscription content. Block crawlers from member areas and rely on logins, not just robots.txt.
- Licensed or exclusive material. Research, datasets or media you sell should not be freely collected.
- Heavy crawl load. If a bot causes real server strain and brings no visitors, block it or rate-limit it at your CDN.
- Private or sensitive pages. Admin areas, internal tools and staging sites should be disallowed and protected.
Common Mistakes That Block AI Visibility
Most AI visibility problems we find come from a few configuration errors.
- **A blanket "Disallow: /" under "User-agent: *".** This silently blocks every crawler without its own group.
- Outdated crawler names. Rules for retired names such as "Claude-Web" do nothing for the current crawlers.
- Firewall blocks. CDN bot protection overrides a correct robots.txt.
- Marketing fluff in llms.txt. Vague claims are not quotable; prices, locations and services are.
- Stale facts. An llms.txt or robots.txt that nobody reviews drifts out of date as your site and the crawlers change.
- Wrong file location. Both files must sit at the domain root; a copy in a subfolder is ignored.
How Nevatrix Fits
Nevatrix configures robots.txt, llms.txt, sitemaps and structured data as part of technical SEO work for businesses across India, from our office in Kazipet, Warangal. We set up the same configuration on our own site before recommending it to clients. We cannot guarantee that any AI tool will cite your business, because each provider decides its own sources; what we can do is make sure nothing technical stops them reading you. See our SEO services, or read how Google AI Overviews choose sources and how to rank in ChatGPT, Gemini and Perplexity.