How to Measure AI Visibility When the Answers Keep Changing
A marketing team asks an AI service which providers a buyer should consider. Their company appears.
A colleague repeats the question and gets a different list.
Which answer should go into the report?
Both, if both meet the same collection rules. The useful question is how often the company appears across a defined set of tests, what the answers say, and whether the evidence supports the next marketing decision.
To measure AI visibility when answers change, fix the buyer questions, repeat the tests, preserve the answers, and report each outcome against the right base. Then use the same rules when you measure again.
1. Decide what you are measuring before you run the test
Start with the audience, product and buying decision you want to understand. A broad question about your industry and a specific comparison question test different situations. Keep the wording consistent so a later change in the result is not simply a change in the question.
Separate questions that name your company from questions that do not. The first tests how AI describes a business it has been asked about. The second tests whether it brings that business into the conversation. Combining them can make discovery look stronger than it is.
Choose the services and surfaces in scope, and label them. A consumer search experience and an API response should not silently become interchangeable observations.
2. Repeat the questions and keep the evidence
Repeat each buyer question on the AI services you want to compare. Keep the question wording and testing conditions consistent across runs.
Repeating a question gives you more evidence than a single answer. A small sample does not establish a precise estimate of what the whole market sees.
Keep the full answer and its cited sources, alongside the question, service, surface, run number and timing. Record the model where it is shown and the relevant location and language settings. A finding should lead back to the answer that supports it.
3. Give each number a clear meaning
A mention rate answers a specific question: in what share of the answers reviewed in scope was the company named?
Mention rate = answers naming the company ÷ answers reviewed in scope.
State that scope beside the number: which buyer questions, services, surfaces and dates it covers. Show results by service as well as in total.
Keep the other outcomes separate:
Recommendation: Does the answer suggest the company as an option to choose? A name in a list is not enough.
Citation: Does the answer point to the company’s website as a source? State which answers support citation measurement.
Accuracy: Do claims about the company match confirmed facts? A citation alone does not establish that they do.
Use an appropriate, disclosed base for each measure. A question subset or a service’s source-reporting limits can change the relevant base.
If a measure was not assessed, label it as unmeasured. A blank field is not evidence of zero performance.
4. Make the retest comparable
Before publishing changes, define what the retest will hold constant: the question wording, service and surface, number of runs, collection rules and classification rules.
Record the publication dates of the work being evaluated. If a model changes, a service is added, or testing conditions differ, disclose it beside the comparison. Where possible, show the shared question-and-service set separately so readers can compare the same scope.
An improvement after publication is an observed change. It does not establish that the content caused it. Read the answers and the sources alongside the numbers before deciding what to do next.
5. Turn the findings into a marketing decision
A useful report should let your team answer three questions:
Where are we consistently absent on questions that matter to the business?
Where are we present but described incorrectly or given little reason to be chosen?
Which content or page change is worth making first, and what will we recheck afterward?
Those are different problems. They may require different work. A single blended score makes it harder to see which action the evidence supports.
Track the next stages separately, too. An AI mention, a website visit, a demo request and a customer are distinct outcomes. Better visibility is useful evidence; sales progress needs its own evidence.
How Signal Harbor puts this into practice
We connect the recorded buyer answers to a prioritized work plan. Your team can inspect the findings, review proposed content and page changes, and approve what gets published. A matched retest then shows what changed within the tested scope.
The goal is a report your marketing team can question, verify and use to decide where to spend its time.
Read our measurement methodology for the definitions and comparison rules.
If you want a starting point for your company, book a complimentary AI Visibility Snapshot and live portal demo. Bring the audience or product you want to focus on, and we will walk through the findings and the next useful step.
Sebastian Miller
Co-Founder, Signal Harbor
signalharborai.com · LinkedIn
This article first appeared in Signal Harbor Weekly, the Signal Harbor newsletter on AI visibility.
Keep reading.
All articlesSignal Harbor, in Plain Answers
If you are looking at Signal Harbor, these are the questions worth starting with.
Ask ChatGPT Who to Hire. Then Look for Your Name.
Imagine a buyer asking ChatGPT, “Which company should we hire for this?” They might be looking for a software provider, a manufacturer, an agency, or a…
What Happens After an AI Visibility Score
When we started Signal Harbor, the first problem was clear: companies had little idea how ChatGPT, Gemini, Perplexity, Google AI Overviews, and other AI…
