August 16, 2026
October 16, 2026
October 16, 2026
September 4, 2026
August 31, 2025
July 31, 2025
June 26, 2025
June 16, 2025
June 8, 2025
May 31, 2025
May 1, 2025
February 6, 2025
November 14, 2024
October 10, 2024
October 1, 2024

How do AI assistants choose which companies to recommend?

Nick Bazuin & Robin Busch
Updated
min read
Summary

In the sources we read, no assistant publishes a ranking formula. ChatGPT says it ranks results using multiple factors and that placement is not guaranteed. What the makers do document is access: Google needs a page to be indexed and snippet-eligible, and OpenAI needs OAI-SearchBot to be able to crawl your site. Those are the parts you control.

  • For Google AI Overviews and AI Mode a page must be indexed and eligible to be shown in Search with a snippet, with no additional technical requirements.
  • Training, search indexing and fetching on request each have their own bot, so one robots.txt rule does not cover everything.
  • User-triggered fetches from ChatGPT-User and Perplexity-User may ignore robots.txt.
  • SE Ranking found no effect of LLMs.txt on how often a domain is cited by LLMs.
  • You can test this today: ask the assistants a recommendation question and open the sources they cite.

When a buyer asks an assistant which agency, supplier or software to use, your company either appears in the answer or it does not. This article explains what the assistant makers say about how they find and use web pages, what you can check on your own site this week and where the evidence runs out. We wrote it for marketing leads and owners of B2B companies, not for SEO specialists. If you want the wider picture first, start with what demonstrably works for being recommended by AI.

No assistant publishes a ranking formula

The makers describe how their assistants find and fetch pages, not how they pick a winner. OpenAI says ChatGPT ranks search results using multiple factors intended to help users find relevant, reliable information, and that placement is not guaranteed. That is why we do not promise rankings in any assistant. What we can do is separate the things that are documented, which are mostly about access, from the things that are guesswork, which is most advice about content tricks.

What the assistants actually do when someone asks

ChatGPT works in two modes, with or without a web search. It may search the web automatically when a question would benefit from current information, and when it does, it typically rewrites the question into one or more targeted queries. The sources we read do not describe how it chooses companies when it answers without searching, so we will not guess.

Google describes AI Overviews and AI Mode in more detail. Both may use a "query fan-out" technique, issuing multiple related searches across subtopics and data sources to build a response. So your page can be pulled in for a question the buyer never typed. The documentation covers AI Overviews and AI Mode, and does not describe the Gemini app separately, so we say nothing about it here.

Perplexity is the most transparent about what you see. It searches the internet in real time and each answer includes numbered citations linking to the original sources. Anthropic's page explains crawling rather than selection, so for Claude we can only describe access. For Microsoft Copilot, the useful fact is that Bing now measures citations, which we come back to below.

One robots.txt rule does not cover every bot

Each company splits its bots by purpose, so one rule cannot cover them all, and they do not all obey robots.txt in the same way. Google's common crawlers always respect robots.txt rules for automatic crawls, and because AI is built into Search, robots.txt directives for Googlebot are the control for how your site is crawled for Search. To limit AI training and grounding in some of Google's other systems, there is Google-Extended.

OpenAI separates the roles explicitly: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training. The catch is user-triggered visits. Because those actions are initiated by a user, robots.txt rules may not apply to ChatGPT-User. Perplexity says the same about its own fetcher: since a user requested the fetch, Perplexity-User generally ignores robots.txt rules. Anthropic is the exception in the sources: its bots respect "do not crawl" signals by honouring industry standard directives in robots.txt.

A typical setup we would recommend for a B2B site that wants to be found looks like this:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /
```

Also check that a firewall is not doing the blocking for you. Perplexity notes that with a Web Application Firewall you may need to explicitly allow its bots, and OpenAI asks you to confirm that your host or content delivery network allows traffic from its published searchbot IP addresses. After a robots.txt change, OpenAI says its systems can take about 24 hours to adjust.

What you give up when you block

Blocking is a trade, and the makers say what it costs. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. At Anthropic, disabling Claude-SearchBot prevents indexing for search and may reduce your visibility and accuracy in user search results, while disabling Claude-User may reduce visibility for user-directed web search. Restricting ClaudeBot is different: it signals that your site's future materials should be excluded from training datasets.

So the choice is not all or nothing. In our experience, a company that wants to be recommended allows the search bots and treats training bots as a separate business decision.

What the evidence says about llms.txt and schema

The research we can point to is limited. In SE Ranking's dataset of nearly 300,000 domains, just 10.13% had an LLMs.txt file. Both statistical analysis and machine learning showed no effect of LLMs.txt on how often a domain is cited by LLMs. The study has a limit: its results reflect the relative importance of factors within the specific model and dataset tested.

For Google, specific optimisation isn't required for AI Overviews and AI Mode, and all existing SEO fundamentals continue to be worthwhile. Schema.org does define a way to point to a page that identifies your organisation, such as its Wikipedia page, Wikidata entry or official website, which is tidy to have. We found no evidence that it changes whether an assistant recommends you, and the same goes for content structure, so we have no tip to invent there.

Check where you stand today

Three checks take under an hour, and none needs a specialist:

  • Open your robots.txt and look for the bots named above, then ask your host whether a firewall filters them.
  • Ask each assistant a question your buyer would ask, then open the sources it cites. OpenAI itself advises opening a cited source to check that it supports the answer and when it was published or updated.
  • Open Bing Webmaster Tools. Its AI Performance report shows how often your content is cited in generative answers, which URLs are referenced and how that changes over time.

Read that last report with care. It shows how often your content is referenced, without indicating placement within an answer, and the data is a sample of overall citation activity. Citation Share is an observational metric and does not represent traffic share. It tells you whether you are being cited, not how many visitors that brings.

Read next

Sources

Free guide

Webflow for B2B, the complete guide as PDF

Costs, migration, SEO and team workflow in one document. Easy to share with your team or management.

What does this mean for your site?

In a 30-minute call we look at your own site and situation. We tell you honestly what we would do, what it would cost and how long it takes. If doing nothing is the better choice, you hear that too.

Nick BazuinRobin Busch
Talk to Nick or Robin
30 minutes · online · free
In 30 minutes you get
An honest view on your own site
A ballpark price and timeline
The next steps, with no strings attached

or mail nick@digitalmakers.nl

Questions

What people ask us about this.

Nick Bazuin

Your question is not here?

Ask Nick directly

01

Can I pay to be recommended by an AI assistant?

01

Can I pay to be recommended by an AI assistant?

The sources we read describe no such option. They describe access and eligibility, and OpenAI states that placement is not guaranteed.

Recommended by AI

01

Does blocking GPTBot stop ChatGPT from recommending me?

01

Does blocking GPTBot stop ChatGPT from recommending me?

Not by itself. GPTBot is the bot for training content, while OAI-SearchBot is the one that lets you appear in search results.

Recommended by AI

01

Do I need special schema markup for Google AI Overviews?

01

Do I need special schema markup for Google AI Overviews?

Google's documentation says there are no additional technical requirements beyond being indexed and snippet-eligible.

Recommended by AI

01

Does an llms.txt file help?

01

Does an llms.txt file help?

SE Ranking found no effect of LLMs.txt on how often a domain is cited by LLMs, in a single dataset of nearly 300,000 domains.

Recommended by AI

Next step

Is your site open to the bots that matter for AI recommendations?

Start with a short check: read your robots.txt, ask two or three buyer questions in the assistants and note whether your company shows up. If you want a second pair of eyes, we are happy to look at it with you, and our overview of [what demonstrably works](/recommended-by-ai) shows where we would spend time first.

New site from

€7,500

Live in

4 to 8 weeks

Proposal within

2 working days

New site in Webflow

Hard to update

From WordPress to Webflow

Next

What should we help with?

Brand

Build

Boost

When would you like to start?

As soon as possible

Within three months

Just exploring

Back

Next

Where can we reach you?

Back

We only use this to reply to your brief.

Thanks, your brief is in.

Nick reads it and replies within one day. Want to talk sooner? Pick a time that suits you.

Your brief

-

Help with

-

Start

-

Your brief did not go through. Please try again in a moment, or book a call with Nick below.

Rather talk it through?

Two men with neutral expressions stand side by side against a dark background.

Book a discovery call