How do large language models choose which websites to reference?
Almost every piece of GEO advice makes sense or falls apart once you understand the pipeline. This is what actually happens between a prompt and a citation.
Almost every piece of advice about AI visibility either makes sense or falls apart once you understand what actually happens between someone typing a question and a citation appearing. The mechanism is not complicated, and knowing it lets you diagnose your own situation rather than working through a generic checklist.
The pipeline
Most generative search systems work in two stages, with a trust assessment running through both.
- Retrieval. The system runs the user’s question against an index — often a live search index plus the model’s own knowledge — and pulls a shortlist of candidate pages.
- Selection and synthesis. The model reads those candidates and chooses which passages to quote and attribute in the answer it writes.
These stages have completely different requirements, and failing them looks identical from the outside. That is why generic advice so often misses: it addresses one stage while your problem is in the other.
Stage 1: retrieval
Retrieval is search. It rewards the things search has always rewarded, plus one requirement that is new.
Indexability and relevance. The page has to be findable and topically matched to the question. This is ordinary SEO, and it is why you cannot be cited from a page that was never retrieved.
Authority. Stronger sites are pulled into candidate sets more readily, exactly as they rank more readily.
Raw-HTML readability. This is the new one, and it is binary. AI crawlers do not execute JavaScript — an analysis by Vercel and MERJ covering more than 500 million GPTBot fetches recorded zero JavaScript execution, with the same behaviour from ClaudeBot and PerplexityBot. If the content only exists after a script runs, the crawler stored an empty shell and there is nothing to retrieve.
Crawler access. Also binary. Blocked at robots.txt or, more often, at the CDN by a bot-management preset nobody chose.
Stage one is an SEO problem with two mechanical additions. Most brands that are absent from AI answers fail here, not at stage two.
Stage 2: selection
Once a page is in the candidate set, the rules change entirely. The model is not choosing a page to send someone to — it is choosing sentences to reproduce.
Four properties decide which passages get used.
Self-containment. A chunk that makes sense without surrounding context is far easier to lift than one that builds across half a page. Passages that depend on a previous paragraph, an unresolved pronoun, or a chart cannot be extracted cleanly.
Directness. The answer stated plainly, early, without preamble. A page that circles for four paragraphs loses to one that answers in two sentences, regardless of which is more thorough overall.
Specificity. A concrete claim can be reproduced without the model inventing anything. A vague one has to be paraphrased, and paraphrase loses attribution. This is the mechanical reason the Princeton and IIT Delhi GEO study found Statistics Addition and Quotation Addition among the highest-performing methods, with roughly 30–40 percent relative visibility improvements.
Fit to the prompt. The passage has to answer the actual question asked, which is usually narrower than the page’s topic. This is why a page answering one question completely beats a page answering six partially.
Find out which stage you are failing
A free scan checks whether AI crawlers can read your pages and shows what the assistants actually say about your category.
Run my free scanThe trust layer
Running through both stages is an assessment of whether a source can be relied on, because a generative answer carries the assistant’s own credibility rather than presenting you with ten options to judge yourself.
That shifts the burden. A search engine can rank a questionable page at position seven and let the user decide. An assistant stating something as fact has to be reasonably confident it is true.
Four things feed that assessment.
Corroboration is the strongest by a distance. Do independent sources describe the same thing the same way? Ten sources agreeing is far stronger evidence than any single claim, however well written. Crucially, unlinked mentions count — reviews, forum threads, roundups and news coverage all contribute, with no hyperlink required.
Self-citation. Pages that name their own sources read as analysis rather than assertion, and a model quoting them inherits those citations as supporting evidence.
Entity clarity. Models reason about entities, not strings. A brand that is unmistakably identifiable, consistently described and clearly placed in a category is easier to state something about than one that describes a mission and never names its market.
Currency. Many systems retrieve from live indexes that favour recency, and a page last touched two years ago is both less likely to be retrieved and less likely to be trusted for a time-sensitive question.
Why answers vary
Ask the same question three times and you may get three different sets of companies named. This is not a malfunction and it has practical consequences.
Generating text involves sampling, so identical prompts produce different outputs by design. The retrieved candidate set can also differ between runs, because live retrieval is not deterministic.
Three things follow. A single answer proves nothing — neither your absence nor your presence. Measurement requires repetition, ideally two or three runs per prompt per engine. And stored evidence matters, because a visibility score without the underlying answers cannot be audited or trended honestly.
Training data vs live retrieval
A common confusion worth resolving, because it determines whether optimisation can work at all.
Models hold general knowledge from training, which is fixed at a point in time and cannot be edited by you. For stable, general questions a model may answer from that alone, with no citations, because there is no retrieval step to cite from.
For anything current, specific, or requiring sources, the system runs live retrieval — and that is the path optimisation affects. It is why changes to your site can influence answers within weeks rather than waiting for a new model generation.
The practical implication: if an assistant answers a question about your category with no citations at all, you are looking at a training-data answer and on-site work will not change it directly. Ask a more specific or more current version of the question and citations usually appear, and that is the surface you can compete on.
Where the engines differ
The pipeline is broadly shared, but the weightings are not, and the differences are large enough to change tactics.
| ChatGPT | Google AI Overviews | Perplexity | |
|---|---|---|---|
| Candidate source | Live search + model knowledge | Google’s index | Live retrieval, heavily |
| Renders JavaScript | No | Yes (Googlebot) | No |
| Weight on organic rank | Moderate | Very high | Moderate |
| Weight on third-party mentions | Very high | Moderate | High |
| Citation transparency | Partial | Explicit | Explicit and prominent |
| Speed of reflecting changes | Slowest | Recrawl cadence | Fastest, often days |
Two practical consequences. Perplexity is the best diagnostic instrument, because it cites openly and updates fast — use it to confirm a page is retrievable at all. And a brand can be strong in AI Overviews on the strength of ordinary SEO while being invisible to ChatGPT for a purely technical reason, which is why measuring only one engine produces confident wrong conclusions.
Four misconceptions about the mechanism
“The model has an opinion about my brand.” Rarely. Absence is nearly always missing or inconsistent information rather than a negative assessment. Models do not dislike you; they have nothing reliable to say.
“If I publish it, the model will learn it.” Training data is fixed at a point in time and you cannot edit it. What you can affect is live retrieval, which is why citations respond to site changes in weeks while general knowledge does not respond at all.
“More content means more citations.” The pipeline selects passages, not sites. Fifty thin pages give a model fifty things not worth quoting. Depth on one question beats coverage across many.
“Schema markup makes me citable.” Structured data supports understanding and attribution; it does not create authority or fix retrieval. Markup on a page nothing can read changes nothing at all.
What this means for you
The mechanism produces a clean diagnostic order.
- Test retrieval first. Ask an assistant something so specific only your page could answer it. If it answers, you are being retrieved and your problem is selection. If it cannot, the problem is mechanical.
- Fix the binary things. Raw-HTML readability and crawler access. Cheap, fast, frequently the whole problem.
- Then fix extractability. Answer-first structure, self-contained passages, specific claims.
- Then work the trust layer. Consistent third-party description, starting with the specific domains the assistants already cite in your category.
Everything in generative engine optimisation reduces to that pipeline. Tactics that map onto it work; tactics that do not — submitting your site, repeating your brand name, publishing volume — do not, and understanding why saves a great deal of wasted effort.
Sources
- Vercel and MERJ — analysis of 500M+ GPTBot fetches finding zero JavaScript execution; same behaviour for ClaudeBot and PerplexityBot.
- Aggarwal, Murahari, Rajpurohit, Kalyan, Narasimhan and Deshpande — “GEO: Generative Engine Optimization”, ACM SIGKDD 2024 (arXiv:2311.09735); ~30–40% relative visibility improvement from statistics, quotations and cited sources across ~10,000 queries.
- OpenAI, Anthropic and Perplexity documentation — crawler identification and robots.txt handling.
Frequently asked questions
How do large language models choose which websites to reference?
Most work in two stages. Retrieval runs the user’s question against a search index and pulls a shortlist of candidate pages, which rewards the same signals as organic ranking. Then selection: the model reads those candidates and quotes the passages that most clearly and credibly answer the prompt. A trust assessment runs through both, weighting sources that other credible sources corroborate.
Do LLMs use their training data or live search?
Both, depending on the question. For stable general knowledge a model may answer from training data alone. For anything current, specific or citable it typically runs a live retrieval step, which is why citations appear on some answers and not others. Optimisation work affects the retrieval path, not the training data, which is why changes can show within weeks rather than waiting for a new model.
Why do AI assistants give different answers to the same question?
Because generating text involves sampling, so identical prompts produce different outputs. The retrieved candidate set can also differ between runs. This is inherent to how the systems work rather than a fault, and it is why a single answer is noise and only a trend across several runs is meaningful.
Does being cited require ranking well on Google?
It strongly helps and is not strictly required. Retrieval leans on organic signals, so a page that ranks well is likely to be in the candidate set. But different engines use different indexes, and a page can be retrieved on the strength of topical relevance and third-party corroboration even without a strong Google position.
What makes a model trust one source over another?
Corroboration is the strongest factor: whether independent sources describe the same thing the same way. Beyond that, whether the page names its own sources, whether claims are specific and dated, whether the site is recognisable as an entity, and whether the content is current. A single self-published claim carries little weight against several agreeing independent ones.
Can I influence what a model was trained on?
Not retrospectively and not directly. What you can influence is what is retrievable now and what the web says about you, both of which feed the live retrieval path that produces citations. Content published today may eventually appear in future training data, but that is a slow, uncontrollable side effect rather than a strategy.