Digital Marketing 11 min read2 July 2026Updated

llms.txt vs robots.txt: AI Crawler Configuration Guide 2026

Add to preferred sources

AI crawlers do not read your site the way Googlebot does — and most robots.txt files still have no idea GPTBot, ClaudeBot or PerplexityBot exist. This is a hands-on technical guide to configuring both files correctly, with Nevatrix's own live setup as a real example.

robots.txt controls which crawlers may access your site, while llms.txt is an optional summary file that points AI systems to your most useful pages. For AI visibility, robots.txt matters most: if it blocks crawlers such as OAI-SearchBot or PerplexityBot, those tools cannot read or cite your pages. This guide shows how to set up both.

Key takeaways

  • robots.txt is a formal standard (RFC 9309) that well-behaved crawlers follow; llms.txt is a voluntary proposal that only some AI tools read.
  • Each AI company runs separate crawlers for training, search and user-requested fetches, so you can allow some and block others.
  • Blocking Google-Extended does not remove you from Google Search or AI Overviews; those use Googlebot.
  • Most robots.txt files were written before AI crawlers existed, so their AI policy is accidental rather than chosen.
  • Check your CDN or firewall too: many sites block AI crawlers there even when robots.txt allows them.

This guide explains what each file does, which AI crawlers exist in 2026, how to write both files, and the configuration running on nevatrix.com today. It is written for site owners and developers who want AI search tools to read their site deliberately rather than by accident.

What Is robots.txt and How Do AI Crawlers Use It?

robots.txt is a plain-text file at the root of your site that tells crawlers which paths they may or may not request. It uses User-agent lines to name a crawler and Allow or Disallow rules to set its access. The format is standardised as the Robots Exclusion Protocol, RFC 9309, and Google explains how it reads the file in its introduction to robots.txt.

AI crawlers read the same file, but each one has its own name. If your robots.txt has no rule for a crawler, the rules under "User-agent: *" apply to it. That is why an old robots.txt written for search engines can unintentionally allow every AI crawler, or block them all.

robots.txt is a request, not a lock. Reputable crawlers follow it, but it does not stop a determined scraper. If you need to keep content private, put it behind a login instead. Also remember that robots.txt is public: anyone can read it, so listing secret paths in it can draw attention to them rather than hide them.

What Is llms.txt and Does Your Site Need One?

llms.txt is a proposed Markdown file at yourdomain.com/llms.txt that gives AI systems a short, structured guide to your site: who you are, what you offer and which pages matter most. It is described at llmstxt.org and is voluntary; it is not an official web standard and no AI company is obliged to read it.

You do not strictly need one. Google says that no special files are required for its AI features and that normal SEO practices apply, as set out in its guidance on AI features and your website. An llms.txt file is still a cheap, low-risk addition. It takes an hour to write, helps AI tools and agents that do read it, and forces you to state your key facts clearly, which also improves your normal pages.

robots.txtllms.txt
PurposeGrants or denies crawler accessSummarises your site for AI systems
StatusFormal standard (RFC 9309)Voluntary community proposal
Who reads itAll reputable crawlersSome AI tools and agents
Effect if missingAll crawlers may access everythingNo effect on access
Risk if wrongCan block search engines or AI tools entirelyCan give AI tools outdated facts

Which AI Crawlers Should You Know About in 2026?

The main AI crawlers come from OpenAI, Anthropic, Perplexity, Google, Apple, Amazon, Meta and Common Crawl, and most companies run separate bots for training and for search or live fetches. The table lists the user-agent names their own documentation publishes.

Crawler (user-agent)OperatorWhat it is used for
GPTBotOpenAICollecting content that may be used to train models
OAI-SearchBotOpenAIIndexing pages for ChatGPT search results
ChatGPT-UserOpenAIFetching a page when a user asks ChatGPT to
ClaudeBotAnthropicCollecting content that may be used to train models
Claude-SearchBotAnthropicIndexing pages to improve search results in Claude
Claude-UserAnthropicFetching a page when a user asks Claude to
PerplexityBotPerplexityIndexing pages for Perplexity answers
Perplexity-UserPerplexityFetching pages in response to a user's question
Google-ExtendedGoogleControlling use of content for Gemini models; not used for Search ranking
Applebot-ExtendedAppleControlling use of content for Apple's AI models
CCBotCommon CrawlBuilding a public web dataset used by many AI developers

Check the official pages before relying on any list, because names change: OpenAI's crawler documentation, Anthropic's crawler policy, Perplexity's crawler guide, Google's list of common crawlers and Apple's Applebot page. Older names such as "anthropic-ai" and "Claude-Web" have been replaced by the Claude crawlers above.

The Accidental Policy Problem

The Accidental Policy Problem is our name for the situation on most websites: nobody ever decided whether AI tools may read the site. The robots.txt was generated years ago by a CMS or plugin, and its AI policy is whatever the "User-agent: *" group happens to say. Fixing it is simple: decide which crawlers you want, then name each one explicitly so your policy is a choice.

Should You Allow or Block AI Crawlers?

Most businesses that want customers to find them should allow AI search and user-fetch crawlers, then decide separately about training crawlers. Being readable by ChatGPT search, Claude and Perplexity gives you a chance to be cited in their answers; blocking them removes that chance.

Your goalSearch and user-fetch crawlersTraining crawlers
Maximum AI visibility for a service businessAllowAllow
Visibility in AI answers, but no training useAllow (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot)Block (GPTBot, ClaudeBot, CCBot)
Protect paid or exclusive contentBlock on paywalled pathsBlock
Stay out of AI tools entirelyBlockBlock, and accept lost visibility

Two cautions apply. First, some providers say user-initiated fetchers may not follow robots.txt in the same way as automated crawlers, so read each provider's documentation. Second, blocking a training crawler does not remove content a model has already learned.

How to Configure robots.txt for AI Crawlers

You configure robots.txt for AI by adding an explicit group for each crawler you want to allow or block, below your general rules. These steps take about half an hour.

Step 1: Read your current file

Open yourdomain.com/robots.txt and note any "Disallow: /" lines, especially under "User-agent: *". A site-wide disallow there blocks every crawler without its own group.

Step 2: Decide your policy

Use the table above to choose which search, fetch and training crawlers you want, and write the decision down so it can be reviewed later.

Step 3: Add explicit groups

Add one group per crawler, or list several user-agents above one shared set of rules. A minimal group that allows AI search and fetch crawlers looks like this:

Example robots.txt group for AI search and fetch crawlers

  • User-agent: OAI-SearchBot
  • User-agent: ChatGPT-User
  • User-agent: Claude-SearchBot
  • User-agent: Claude-User
  • User-agent: PerplexityBot
  • User-agent: Perplexity-User
  • Allow: /
  • Disallow: /admin/

Step 4: Keep the sitemap line

End the file with a Sitemap line pointing to your XML sitemap, so every crawler can find your pages.

Step 5: Check your CDN and firewall

Bot-protection settings at your host or CDN can block AI crawlers even when robots.txt allows them. Check the security settings and logs for blocked requests from the crawlers you allowed.

Step 6: Test and review

Use Google Search Console's robots.txt report to confirm the file is valid, then review the policy every few months as new crawlers appear.

A copyable checklist version:

  1. 1Current robots.txt read and site-wide blocks noted.
  2. 2Policy for search, fetch and training crawlers written down.
  3. 3Explicit groups added for each chosen crawler.
  4. 4Sitemap line present at the end of the file.
  5. 5CDN and firewall bot settings checked.
  6. 6File validated and a review date set.

How to Write an llms.txt File

An llms.txt file is a short Markdown document with your name, a one-line summary, and grouped links to your most important pages, each with a brief description. Keep it factual: AI systems quote specifics such as services, locations and prices, not adjectives.

  • First line: a level-one heading with your business name.
  • Summary: one blockquote sentence saying what you do and for whom.
  • Key facts: location, year founded, main services and starting prices that match your website exactly.
  • Sections of links: for example Services, Guides and Company, each a list of links with one-line descriptions.
  • Optional section: lower-priority links that AI tools can skip when they have limited space.

Example llms.txt structure for a service business

  • # Your Business Name
  • > One sentence on what you do, for whom and where.
  • Key facts: city, year founded, main services, starting prices.
  • ## Services: links to each service page with a one-line description.
  • ## Guides: links to your most useful articles with one-line descriptions.
  • ## Company: about, contact and pricing pages.
  • ## Optional: lower-priority pages AI tools can skip.

Which pages should llms.txt link to?

Link to the pages you would want an AI tool to quote: service pages with prices, location pages, your most complete guides, your about page and your contact page. Leave out thin tag pages, old news posts and anything you would not want summarised. Twenty to forty well-described links is plenty for most business sites; a list of every URL defeats the purpose, which is to point AI tools at your best material first.

Update llms.txt whenever prices, services or key pages change. An out-of-date summary is worse than none, because AI tools may repeat its old facts. If you write about AI search more broadly, our guide to generative engine optimization explains how these files fit into a wider visibility plan.

Does robots.txt Affect Google AI Overviews?

Yes, but only through Googlebot: if Googlebot can crawl and index a page, it can appear in Google's AI Overviews, and if it cannot, it cannot. Google-Extended has no effect on Search or AI Overviews. Google also offers snippet controls, such as the nosnippet meta tag, that limit how much of a page can be shown, and these apply to AI features as well as normal results. If you want to understand how Google picks which pages to cite, our guide to getting cited in Google AI Overviews covers the content side.

How Do You Know If AI Tools Are Reading Your Site?

You can tell AI tools are reading your site from two places: crawler visits in your server or CDN logs, and visitors arriving from AI tools in your analytics. Checking both every month shows whether your configuration is working.

  • Server and CDN logs: filter requests by user-agent for names such as OAI-SearchBot, ClaudeBot or PerplexityBot. Regular visits mean the crawler can reach you; blocked or error responses mean something is stopping it.
  • Referral traffic: in your analytics, look for visits referred from chatgpt.com, perplexity.ai, claude.ai and gemini.google.com. These are people who clicked a link in an AI answer.
  • Manual checks: ask the AI tools questions your customers would ask and note whether your business or pages are mentioned.
  • Search Console: AI Overviews traffic is included in your normal Google Search performance data.

If crawlers visit but you never appear in answers, the problem is usually content rather than configuration. Pages need specific, quotable facts and a clear structure; our comparison of SEO, GEO and AEO explains what each discipline adds.

What Else Helps AI Tools Understand Your Site?

Beyond robots.txt and llms.txt, AI tools rely on the same signals as search engines: an up-to-date sitemap, structured data, fast server-rendered pages and clear, specific writing. These usually matter more than llms.txt itself.

  • XML sitemap: lists every page you want found and when it last changed, so crawlers spend their time on current pages.
  • Structured data: schema markup such as Organization, LocalBusiness, FAQPage and Article tells machines exactly what a page is about; Google's introduction to structured data explains the formats.
  • Server-rendered content: many AI crawlers do not run JavaScript, so text that only appears after scripts load may be invisible to them.
  • Consistent facts: the same name, address, prices and services on your website, Google Business Profile and directories make it easier for AI tools to trust what they read.
  • Quotable passages: short paragraphs that answer one question each are easier for AI tools to lift into an answer than long, mixed sections.

If a crawler can reach your site but your pages are slow, script-heavy or vague, it will read them and still choose other sources. Configuration opens the door; content decides whether you are cited.

Real Example: The robots.txt and llms.txt on nevatrix.com

nevatrix.com allows major search engines and AI crawlers explicitly and publishes an llms.txt file. Our robots.txt has a general group, a group for search engines, and a separate group naming AI crawlers including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended and CCBot, each allowed with only private paths such as /api/ and /admin/ disallowed. A few aggressive scrapers that bring no visitors are blocked.

Our llms.txt lists our services, locations and main guides with one-line descriptions. We chose to allow training crawlers as well as search crawlers, because for a service business being understood by AI tools is worth more than keeping public marketing pages out of training data. A publisher selling content might sensibly decide the opposite.

When Allowing AI Crawlers Is Not Worth It

Blocking AI crawlers is the right call when your content is your product, or when crawling costs outweigh any visibility benefit.

  • Paywalled or subscription content. Block crawlers from member areas and rely on logins, not just robots.txt.
  • Licensed or exclusive material. Research, datasets or media you sell should not be freely collected.
  • Heavy crawl load. If a bot causes real server strain and brings no visitors, block it or rate-limit it at your CDN.
  • Private or sensitive pages. Admin areas, internal tools and staging sites should be disallowed and protected.

Common Mistakes That Block AI Visibility

Most AI visibility problems we find come from a few configuration errors.

  • **A blanket "Disallow: /" under "User-agent: *".** This silently blocks every crawler without its own group.
  • Outdated crawler names. Rules for retired names such as "Claude-Web" do nothing for the current crawlers.
  • Firewall blocks. CDN bot protection overrides a correct robots.txt.
  • Marketing fluff in llms.txt. Vague claims are not quotable; prices, locations and services are.
  • Stale facts. An llms.txt or robots.txt that nobody reviews drifts out of date as your site and the crawlers change.
  • Wrong file location. Both files must sit at the domain root; a copy in a subfolder is ignored.

How Nevatrix Fits

Nevatrix configures robots.txt, llms.txt, sitemaps and structured data as part of technical SEO work for businesses across India, from our office in Kazipet, Warangal. We set up the same configuration on our own site before recommending it to clients. We cannot guarantee that any AI tool will cite your business, because each provider decides its own sources; what we can do is make sure nothing technical stops them reading you. See our SEO services, or read how Google AI Overviews choose sources and how to rank in ChatGPT, Gemini and Perplexity.

Get an AI Crawler Configuration Audit

Related Articles & Services

Integrating LLMs into Your Existing Software →AI Search Optimization (GEO) — Complete Guide 2026 →How to Rank in ChatGPT, Gemini and Perplexity →Get an AI Crawler Audit →
RB

About the Author

Rathan Babu Since 2012 experience

Rathan Babu is the Founder and Lead Developer at Nevatrix Technologies, Warangal. Building software since 2012, he has personally built 100+ web applications, SaaS platforms and AI-powered systems for businesses across India, the USA, Canada and the UK.

View full profile → LinkedIn

Frequently Asked Questions

What is the difference between llms.txt and robots.txt?

robots.txt controls which crawlers may access your pages and is a formal standard. llms.txt is a voluntary Markdown summary that points AI tools to your key pages; it does not control access.

Will allowing AI crawlers hurt my Google rankings?

No. Google Search rankings are handled by Googlebot, and allowing or blocking AI crawlers such as GPTBot or PerplexityBot does not change how Google ranks your pages.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls use of your content for Gemini models. Google Search and AI Overviews rely on Googlebot, so blocking Google-Extended does not remove you from Search.

Should I block GPTBot to stop OpenAI training on my content?

Only if that matters more than visibility. Blocking GPTBot stops future training collection, while OAI-SearchBot and ChatGPT-User can still be allowed for ChatGPT search and user fetches.

Where do I put the llms.txt file?

At the root of your domain, so it loads at yourdomain.com/llms.txt, next to robots.txt. Serve it as plain text or Markdown and keep it publicly accessible.

Does llms.txt guarantee I will be cited by ChatGPT or Perplexity?

No. It is voluntary and only some tools read it. Citations depend mainly on being crawlable, ranking well and having clear, specific content.

How do I check whether my robots.txt blocks AI crawlers?

Open yourdomain.com/robots.txt and look for a "Disallow: /" under "User-agent: *" or under named AI crawlers. Then check your CDN or firewall logs for blocked bot requests.

How often should I update robots.txt for AI crawlers?

Every few months. AI companies add and rename crawlers, so review each provider's documentation and update your groups when names change.

Ready to Grow Your Business?

Contact Nevatrix, a web development and digital marketing company in Warangal, for a free consultation and quote.