Why Does ChatGPT Give Different Site Selection Shortlists to the Same Question?
Michael Etheredge is VP of Product Development at QuestionFuel, where he leads how the firm measures and corrects the way AI answer engines describe companies. He was featured by the Southern Economic Development Council on how AI is changing the way companies research communities and regions.
Ask an answer engine the same location question twice and you may get two different shortlists. This article covers what the research documents, what it does not explain, and a simple way for economic developers to measure answer consistency.
Key takeaways
- Published tests show location shortlists differ across engines and across repeat runs of the same question.
- The studies document the variation but do not isolate its causes, so treat any single result as one sample.
- Answer consistency can be measured by running the same prompt five times and classifying the region as consistent, frequent, occasional, or absent.
- Report ranges with dates and engine names, not one screenshot.
Quick answer: Answer engines can return different location shortlists to the same question, and the variation shows up across engines and across repeat runs. The Site Selectors Guild saw it in its own test, and a 2026 study found one brand's visibility ranged from 15.5% to 59.5% by engine. Report ranges, not single results.
What did the Site Selectors Guild test find?
The Site Selectors Guild and Development Counsellors International tested three answer engines on two location scenarios: a software company headquarters relocation and a manufacturing capacity expansion. The engines were an early Google answer engine and two versions of ChatGPT.
According to the commentary published in Site Selection Magazine in November 2024, the engines' shortlists overlapped with each other and with human generated lists, but they also deviated, and recommendations varied by engine. The authors also reported that asking the same engine the same question multiple times returned different versions of the shortlist.
That second finding matters most for economic developers. It means one test is one sample.
Is shortlist variation a defect or normal behavior?
The studies we found document the variation. They do not isolate its causes, so we cannot say how much comes from the engine, the sources it retrieves, or chance.
One useful clue comes from Zen Media, an AI visibility agency. In its September 2026 analysis of three campaigns and 900 buyer intent prompts, the same company appeared in 15.5% to 59.5% of prompts depending on the engine, even on identical prompts. The source mix differed too. In one campaign a wire release was cited slightly more often than the client's own site, while in another the client's own website led.
The practical conclusion is that variation is normal enough to plan for. It is not evidence that a single result is wrong, only that a single result is incomplete.
How can an economic development organization measure answer consistency?
Answer consistency: how often a region appears across repeated runs of the same question, in new chats, on the same engine.
Run each shortlist prompt five times in new, logged out chats and classify the result.
| Class | Runs where your region appeared | What it suggests |
|---|---|---|
| Consistent | 5 of 5 | The region is reliably included for that question |
| Frequent | 3 or 4 of 5 | Included most of the time, but not dependable |
| Occasional | 1 or 2 of 5 | A possible case of Regional Silent Exclusion that a single run could miss or exaggerate |
| Absent | 0 of 5 | Strong signal of Regional Silent Exclusion for that question |
These are working categories QuestionFuel suggests for planning. They are not an industry standard, and five runs is a minimum, not a statistical guarantee. Our step by step audit guide covers the full prompt set.
How should you report results to a board or members?
Report ranges, with dates and engine names. A single screenshot can look either alarming or reassuring by chance.
A useful line for a board update looks like this: "On October 6, in logged out chats, our region appeared in 3 of 5 runs of the manufacturing shortlist question on ChatGPT, and two competitor regions appeared in 5 of 5." It states the date, the engine, the method, and the comparison, and it can be repeated next quarter.
What limits apply to this evidence?
Three limits matter.
- •Model age. The Guild test used 2023 models. Current engines behave differently, so use the finding as a reason to test, not as a measure of today's rates.
- •Scope. The Guild tested two scenarios. Zen Media measured company visibility, not regions, and is an agency reporting its own campaigns.
- •Cause. Neither source explains why answers vary, so avoid any claim about the mechanism.
Where does this fit in a regional audit?
Variation is why the audit repeats every prompt. It also connects to accuracy: AI Narrative Drift can show up on one run and not the next. Start with How Does AI Describe Your Region to Companies Deciding Where to Build? for the broader evidence, then run the audit.
Frequently asked questions
Is one screenshot of ChatGPT enough evidence about my region?
No. Published tests show the same question can return different shortlists, so one screenshot is one sample. Run each prompt several times and report the range.
How many runs do I need?
Five per prompt is a sensible minimum for a quick test. More runs give a steadier picture, and the number should be stated whenever results are shared.
Do different answer engines give the same shortlist?
Not reliably. The Guild test found recommendations varied by engine, and Zen Media found one company's visibility varied widely by engine. Test each engine your audience uses and keep results separate.
Can an economic development organization reduce the variation?
Possibly, by making accurate facts about the region consistent and easy to find across sources. No one can guarantee a result, and the effect should be measured by repeating the test rather than assumed.
Sources and references
- •Site Selectors Guild, The Potential and Limitations of AI in Site Selection
- •Site Selection Magazine, AI Challenges Traditional Site Selection, November 2024
- •Zen Media, One Brand's AI Visibility Tracking Ranged From 15.5% to 59.5%, September 2026
Related reading: How Does AI Describe Your Region to Companies Deciding Where to Build? · How to Audit What AI Says About Your Region · Silent Exclusion: Why Qualified AEC Firms Disappear From AI Search Results · Does AI Describe Your Region by the Criteria Site Selectors Now Use? A 2026 Checklist · How Should an Economic Development Organization Choose an AI Visibility Program? · How Do Answer Engines Describe Regions? A Guide for Economic Development Organizations