When a buyer asks an assistant which agency, supplier or software to use, your company either appears in the answer or it does not. This article explains what the assistant makers say about how they find and use web pages, what you can check on your own site this week and where the evidence runs out. We wrote it for marketing leads and owners of B2B companies, not for SEO specialists. If you want the wider picture first, start with what demonstrably works for being recommended by AI.
No assistant publishes a ranking formula
The makers describe how their assistants find and fetch pages, not how they pick a winner. OpenAI says ChatGPT ranks search results using multiple factors intended to help users find relevant, reliable information, and that placement is not guaranteed. That is why we do not promise rankings in any assistant. What we can do is separate the things that are documented, which are mostly about access, from the things that are guesswork, which is most advice about content tricks.
What the assistants actually do when someone asks
ChatGPT works in two modes, with or without a web search. It may search the web automatically when a question would benefit from current information, and when it does, it typically rewrites the question into one or more targeted queries. The sources we read do not describe how it chooses companies when it answers without searching, so we will not guess.
Google describes AI Overviews and AI Mode in more detail. Both may use a "query fan-out" technique, issuing multiple related searches across subtopics and data sources to build a response. So your page can be pulled in for a question the buyer never typed. The documentation covers AI Overviews and AI Mode, and does not describe the Gemini app separately, so we say nothing about it here.
Perplexity is the most transparent about what you see. It searches the internet in real time and each answer includes numbered citations linking to the original sources. Anthropic's page explains crawling rather than selection, so for Claude we can only describe access. For Microsoft Copilot, the useful fact is that Bing now measures citations, which we come back to below.
One robots.txt rule does not cover every bot
Each company splits its bots by purpose, so one rule cannot cover them all, and they do not all obey robots.txt in the same way. Google's common crawlers always respect robots.txt rules for automatic crawls, and because AI is built into Search, robots.txt directives for Googlebot are the control for how your site is crawled for Search. To limit AI training and grounding in some of Google's other systems, there is Google-Extended.
OpenAI separates the roles explicitly: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. The catch is user-triggered visits. Because those actions are initiated by a user, robots.txt rules may not apply to ChatGPT-User. Perplexity says the same about its own fetcher: since a user requested the fetch, Perplexity-User generally ignores robots.txt rules. Anthropic is the exception in the sources: its bots respect "do not crawl" signals by honouring industry standard directives in robots.txt.
A typical setup we would recommend for a B2B site that wants to be found looks like this:
```
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
```
Also check that a firewall is not doing the blocking for you. Perplexity notes that with a Web Application Firewall you may need to explicitly allow its bots, and OpenAI asks you to confirm that your host or content delivery network allows traffic from its published searchbot IP addresses. After a robots.txt change, OpenAI says its systems can take about 24 hours to adjust.
What you give up when you block
Blocking is a trade, and the makers say what it costs. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. At Anthropic, disabling Claude-SearchBot prevents indexing for search and may reduce your visibility and accuracy in user search results, while disabling Claude-User may reduce visibility for user-directed web search. Restricting ClaudeBot is different: it signals that your site's future materials should be excluded from training datasets.
So the choice is not all or nothing. In our experience, a company that wants to be recommended allows the search bots and treats training bots as a separate business decision.
What the evidence says about llms.txt and schema
The research we can point to is limited. In SE Ranking's dataset of nearly 300,000 domains, just 10.13% had an LLMs.txt file. Both statistical analysis and machine learning showed no effect of LLMs.txt on how often a domain is cited by LLMs. The study has a limit: its results reflect the relative importance of factors within the specific model and dataset tested.
For Google, specific optimisation isn't required for AI Overviews and AI Mode, and all existing SEO fundamentals continue to be worthwhile. Schema.org does define a way to point to a page that identifies your organisation, such as its Wikipedia page, Wikidata entry or official website, which is tidy to have. We found no evidence that it changes whether an assistant recommends you, and the same goes for content structure, so we have no tip to invent there.
Check where you stand today
Three checks take under an hour, and none needs a specialist:
- Open your robots.txt and look for the bots named above, then ask your host whether a firewall filters them.
- Ask each assistant a question your buyer would ask, then open the sources it cites. OpenAI itself advises opening a cited source to check that it supports the answer and when it was published or updated.
- Open Bing Webmaster Tools. Its AI Performance report shows how often your content is cited in generative answers, which URLs are referenced and how that changes over time.
Read that last report with care. It shows how often your content is referenced, without indicating placement within an answer, and the data is a sample of overall citation activity. Citation Share is an observational metric and does not represent traffic share. It tells you whether you are being cited, not how many visitors that brings.



