AI crawlers, robots.txt and llms.txt: a practical guide for businesses that want to be found in AI answers
AI assistants find and cite websites through their own crawlers and search indexes. This guide lists the main AI user agents, explains search versus training crawlers, shows robots.txt examples, and covers llms.txt, CDN bot blocking and how to check access.
AI assistants such as ChatGPT, Claude, Gemini and Perplexity find web pages through their own crawlers and through search indexes. To be found and cited, allow their search and user-triggered crawlers in robots.txt, make sure your CDN or firewall is not silently blocking them, and give them clean, fast HTML. Training crawlers are a separate decision.
Many Indian business sites block AI crawlers without knowing it, through an over-cautious security plugin, a CDN "block AI bots" toggle or a copied robots.txt. Others let everything in but serve content only through JavaScript that crawlers struggle to read. This guide follows the AI search module of SEO & AI Search Mastery.
Which AI crawlers should you know?
Each company documents its crawlers: see OpenAI’s bot documentation and Google’s list of crawlers. User agent names change occasionally, so check the official pages once a quarter.
What is the difference between search and training crawlers?
Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) decide whether you can appear and be cited in AI search answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) collect content to train future models. Blocking a training crawler does not usually remove you from that company’s search results, and blocking a search crawler does. A business that wants visibility should allow search crawlers; whether to allow training is a content-licensing choice.
What should robots.txt look like if you want maximum AI visibility?
User-agent: * Allow: / # Explicitly welcome AI search and assistant crawlers User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://www.example.in/sitemap.xml
A blanket User-agent: * with Allow: / already permits every compliant crawler. Named groups make your intent explicit and protect against later edits. Remember that if a crawler has its own named group, it follows only that group, so repeat any Disallow rules (for example for admin or cart pages) inside named groups too.
Why might AI crawlers still be blocked?
Test, do not assume. Fetch your pages with the crawler’s user agent from a terminal (for example with curl) and confirm you get a 200 response with the real content, then check server logs for visits from these user agents.
What is llms.txt and should you have one?
llms.txt is a proposed convention: a plain-text Markdown file at your site root that summarises who you are and links to your most useful pages, so AI tools can understand the site quickly. It is not an official standard and no major AI search engine has confirmed using it for ranking. It is cheap to create, harmless and useful for AI agents and developers, so many sites add one. Keep it factual: what you do, key pages, products or courses with prices, contact details and important facts such as policies.
careersninza.com publishes llms.txt with programmes, fees and key facts, and a longer llms-full.txt with full syllabi, both generated from the same data as the site so they never drift out of date.
How do you check whether AI systems can see and cite you?
Source: Anthropic Economic Index India brief (February 2026). Indian AI assistant use is large and concentrated in these states.
Should you block AI crawlers to protect your content?
It depends on the business. Publishers whose revenue depends on page views or licensing may restrict training crawlers. Most businesses that sell services, products or courses benefit from being described accurately by AI assistants, so they allow search crawlers and often training crawlers too. Whatever you decide, decide deliberately and document it.
What are the limits of crawler access?
Allowing crawlers makes you eligible; it does not guarantee citations. AI systems still choose sources based on relevance, clarity, authority and freshness. Access is the first step; useful, specific content and consistent entity signals do the rest.
How do you test crawler access from the command line?
A quick test needs no special tools. From a terminal, request a page with the crawler’s user agent and check the status code and content. For example:
curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://www.example.in/
curl -s -A "Claude-SearchBot" https://www.example.in/services/ | grep -i "<h1"A 200 status and your real headline mean the request got through. A 403, a challenge page or an empty body means something in front of your site (CDN, firewall or security plugin) is blocking that agent, whatever robots.txt says. Repeat for your most important page types: home, a service or product page, a blog post and the sitemap.
What should a good llms.txt contain?
Generate it from the same data that powers your site if you can, so prices and facts never contradict the pages. A stale llms.txt that lists last year’s fees is worse than none.
Which server and site settings help AI crawlers?
These are ordinary technical SEO practices, which is the point: the sites that are easiest for Googlebot to crawl are usually easiest for AI crawlers too.
Our SEO & AI Search Mastery batches run live online, so the same class reaches learners and business owners in metros such as Bengaluru, Pune, Kolkata, Mumbai and Delhi NCR, growing cities such as Jaipur, Kanpur, Madurai, Raipur and Dehradun, and smaller industrial towns such as Panipat, Firozabad, Karur, Durgapur and Dibrugarh, at the same fee. Each city page explains how local businesses use AI search visibility, for example in Jaipur, Panipat, Madurai.
Learn this live: SEO & AI Search Mastery
Configure crawler access, robots.txt, llms.txt and structured data, then measure AI citations for a real site. 20 live hours, ₹19,999, single payment.
Rank on Google and get cited by AI answer engines.
Frequently asked questions
Related reading
We teach this, live
Every article here comes from something we teach. Sit in on a free masterclass and judge the mentors yourself.