PROGRAM ZERO 18-month live AI, LLM & full-stack programme · ₹89,999 or ₹7,500/month · Starts 9 January 2027
Explore Program Zero →
Careers Ninza — business and startup leadership training
JOIN ZERO NINZA KIDS
JOIN ZERO NINZA KIDS
See all programs → Free live masterclass → Guides & articles → Newsletter →
Careers Ninza
SEO & AI SEARCH 7 min read · Updated 28 October 2026

AI crawlers, robots.txt and llms.txt: a practical guide for businesses that want to be found in AI answers

AI assistants find and cite websites through their own crawlers and search indexes. This guide lists the main AI user agents, explains search versus training crawlers, shows robots.txt examples, and covers llms.txt, CDN bot blocking and how to check access.

CN
Careers Ninza SEO faculty
Careers Ninza · Kolkata, India
Learn this live in our SEO & AI Search Mastery course

AI assistants such as ChatGPT, Claude, Gemini and Perplexity find web pages through their own crawlers and through search indexes. To be found and cited, allow their search and user-triggered crawlers in robots.txt, make sure your CDN or firewall is not silently blocking them, and give them clean, fast HTML. Training crawlers are a separate decision.

Many Indian business sites block AI crawlers without knowing it, through an over-cautious security plugin, a CDN "block AI bots" toggle or a copied robots.txt. Others let everything in but serve content only through JavaScript that crawlers struggle to read. This guide follows the AI search module of SEO & AI Search Mastery.

Which AI crawlers should you know?

CompanyUser agentPurpose
OpenAIOAI-SearchBotIndexes pages for ChatGPT search results
OpenAIChatGPT-UserFetches a page when a user asks ChatGPT to
OpenAIGPTBotCollects content for model training
AnthropicClaude-SearchBotIndexes pages for Claude’s search
AnthropicClaude-UserFetches pages for user requests
AnthropicClaudeBotCollects content for model training
PerplexityPerplexityBot / Perplexity-UserSearch index / user-requested fetches
GoogleGooglebot / Google-ExtendedSearch (incl. AI Overviews) / Gemini training and grounding control
AppleApplebot / Applebot-ExtendedSearch features / training control
MicrosoftBingbotBing index, also used by Copilot and others
OthersAmazonbot, Meta-ExternalAgent, Bytespider, CCBot, DuckAssistBot, MistralAI-UserVarious assistants, training and datasets

Each company documents its crawlers: see OpenAI’s bot documentation and Google’s list of crawlers. User agent names change occasionally, so check the official pages once a quarter.

What is the difference between search and training crawlers?

Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) decide whether you can appear and be cited in AI search answers. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) collect content to train future models. Blocking a training crawler does not usually remove you from that company’s search results, and blocking a search crawler does. A business that wants visibility should allow search crawlers; whether to allow training is a content-licensing choice.

What should robots.txt look like if you want maximum AI visibility?

User-agent: *
Allow: /

# Explicitly welcome AI search and assistant crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://www.example.in/sitemap.xml

A blanket User-agent: * with Allow: / already permits every compliant crawler. Named groups make your intent explicit and protect against later edits. Remember that if a crawler has its own named group, it follows only that group, so repeat any Disallow rules (for example for admin or cart pages) inside named groups too.

Why might AI crawlers still be blocked?

—CDN or firewall settings: Cloudflare and other providers offer one-click AI bot blocking and "managed robots.txt" features. Check them; a dashboard toggle can override your intentions.
—Security plugins that block unknown user agents or rate-limit aggressively.
—Country or bot-fight rules that challenge requests from data centres abroad.
—JavaScript-only content that many crawlers do not render.
—Login walls and pop-ups that hide the content.

Test, do not assume. Fetch your pages with the crawler’s user agent from a terminal (for example with curl) and confirm you get a 200 response with the real content, then check server logs for visits from these user agents.

What is llms.txt and should you have one?

llms.txt is a proposed convention: a plain-text Markdown file at your site root that summarises who you are and links to your most useful pages, so AI tools can understand the site quickly. It is not an official standard and no major AI search engine has confirmed using it for ranking. It is cheap to create, harmless and useful for AI agents and developers, so many sites add one. Keep it factual: what you do, key pages, products or courses with prices, contact details and important facts such as policies.

careersninza.com publishes llms.txt with programmes, fees and key facts, and a longer llms-full.txt with full syllabi, both generated from the same data as the site so they never drift out of date.

How do you check whether AI systems can see and cite you?

1Check robots.txt and CDN bot settings.
2Fetch key pages with each user agent and confirm the full content is returned.
3Search your server logs for AI user agents and look at which pages they fetch.
4Ask ChatGPT search, Claude, Perplexity and Gemini specific questions your pages answer, and record citations monthly.
5Submit your sitemap to Bing Webmaster Tools; Bing’s index powers several AI products. Use IndexNow for faster discovery of new URLs.
Share of Claude.ai use by Indian state (top four)
Maharashtra15.5%
Tamil Nadu13.2%
Karnataka12.7%
Delhi10.5%

Source: Anthropic Economic Index India brief (February 2026). Indian AI assistant use is large and concentrated in these states.

Should you block AI crawlers to protect your content?

It depends on the business. Publishers whose revenue depends on page views or licensing may restrict training crawlers. Most businesses that sell services, products or courses benefit from being described accurately by AI assistants, so they allow search crawlers and often training crawlers too. Whatever you decide, decide deliberately and document it.

What are the limits of crawler access?

Allowing crawlers makes you eligible; it does not guarantee citations. AI systems still choose sources based on relevance, clarity, authority and freshness. Access is the first step; useful, specific content and consistent entity signals do the rest.

How do you test crawler access from the command line?

A quick test needs no special tools. From a terminal, request a page with the crawler’s user agent and check the status code and content. For example:

curl -s -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://www.example.in/
curl -s -A "Claude-SearchBot" https://www.example.in/services/ | grep -i "<h1"

A 200 status and your real headline mean the request got through. A 403, a challenge page or an empty body means something in front of your site (CDN, firewall or security plugin) is blocking that agent, whatever robots.txt says. Repeat for your most important page types: home, a service or product page, a blog post and the sitemap.

What should a good llms.txt contain?

1A one-line title with your business name.
2A short summary: who you are, where you operate, what you offer, founding year.
3Key facts in plain sentences: prices or price ranges, service areas, policies, contact details.
4Links to your most useful pages, grouped by type, with one-line descriptions.
5Optionally, a link to a fuller file with detailed content, such as llms-full.txt.

Generate it from the same data that powers your site if you can, so prices and facts never contradict the pages. A stale llms.txt that lists last year’s fees is worse than none.

Which server and site settings help AI crawlers?

—Return correct status codes, including real 404s for missing pages.
—Keep the sitemap complete and submit it to both Google and Bing; use IndexNow for Bing, Yandex and other participating engines.
—Serve main content in HTML rather than loading it only through client-side JavaScript.
—Avoid aggressive rate limits that block bursts of legitimate crawling.
—Keep page weight low so fetches complete quickly.

These are ordinary technical SEO practices, which is the point: the sites that are easiest for Googlebot to crawl are usually easiest for AI crawlers too.

Our SEO & AI Search Mastery batches run live online, so the same class reaches learners and business owners in metros such as Bengaluru, Pune, Kolkata, Mumbai and Delhi NCR, growing cities such as Jaipur, Kanpur, Madurai, Raipur and Dehradun, and smaller industrial towns such as Panipat, Firozabad, Karur, Durgapur and Dibrugarh, at the same fee. Each city page explains how local businesses use AI search visibility, for example in Jaipur, Panipat, Madurai.

Learn this live: SEO & AI Search Mastery

Configure crawler access, robots.txt, llms.txt and structured data, then measure AI citations for a real site. 20 live hours, ₹19,999, single payment.

SHORT-TERM COURSE · LIVE ONLINE
SEO & AI Search Mastery

Rank on Google and get cited by AI answer engines.

₹19,999 · single payment20 hoursLive, mentor-ledCertificate on completion

Frequently asked questions

Which AI crawlers should I allow?+

To appear in AI search answers, allow search and user-triggered crawlers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot. Allowing training crawlers such as GPTBot, ClaudeBot and Google-Extended is a separate content decision.

Does blocking GPTBot remove my site from ChatGPT search?+

No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and user-requested fetches use ChatGPT-User. Blocking GPTBot affects training, not search visibility.

What is llms.txt?+

llms.txt is a proposed plain-text file at the root of a website that summarises the site and links to key pages for AI tools. It is not an official standard and its effect on rankings is unconfirmed, but it is easy to create and harmless.

Can Cloudflare block AI crawlers on my site?+

Yes. Cloudflare offers settings that block AI bots or manage robots.txt for you. If you want AI visibility, check those settings and your firewall rules in addition to robots.txt.

How do I know if AI crawlers visit my website?+

Check your server or CDN logs for their user agent names, and fetch your pages using those user agents to confirm they receive the full content with a 200 status.

Where can I learn AI search optimisation?+

Careers Ninza SEO & AI Search Mastery covers technical SEO, structured data and getting cited by AI answer engines on a real site. It is 20 live hours for ₹19,999, online across India.

Related reading

SEO & AI SEARCH SEO Reporting With Search Console and GA4: What to Measure and How to Explain It Good SEO reports connect rankings and traffic to leads and revenue, separate brand from non-brand, and explain changes honestly, including AI search effects. Here is a reporting framework with Search Console, GA4 and Looker Studio. 6 min read SEO & AI SEARCH Ecommerce SEO for Indian Stores: Category Pages, Product Pages, Filters and AI Shopping Ecommerce SEO wins traffic that does not need ad spend. This guide covers category pages, product pages, filter and duplicate control, structured data, Merchant Center free listings and AI shopping answers for Shopify and WooCommerce stores. 6 min read SEO & AI SEARCH Core Web Vitals and Page Speed for Indian Websites: LCP, INP and CLS Fixes That Work Most Indians browse on mid-range Android phones over mobile data, so speed is a ranking and revenue issue. This guide explains Core Web Vitals and the fixes that matter most for WordPress, Shopify and custom sites. 6 min read

We teach this, live

Every article here comes from something we teach. Sit in on a free masterclass and judge the mentors yourself.