Signal Harbor

How to Measure AI Visibility When the Answers Keep Changing

A marketing team asks an AI service which providers a buyer should consider. Their company appears.

Signal Harbor4 min read

A colleague repeats the question and gets a different list.

Which answer should go into the report?

Both, if both meet the same collection rules. The useful question is how often the company appears across a defined set of tests, what the answers say, and whether the evidence supports the next marketing decision.

To measure AI visibility when answers change, fix the buyer questions, repeat the tests, preserve the answers, and report each outcome against the right base. Then use the same rules when you measure again.

1. Decide what you are measuring before you run the test

Start with the audience, product and buying decision you want to understand. A broad question about your industry and a specific comparison question test different situations. Keep the wording consistent so a later change in the result is not simply a change in the question.

Separate questions that name your company from questions that do not. The first tests how AI describes a business it has been asked about. The second tests whether it brings that business into the conversation. Combining them can make discovery look stronger than it is.

Choose the services and surfaces in scope, and label them. A consumer search experience and an API response should not silently become interchangeable observations.

2. Repeat the questions and keep the evidence

Repeat each buyer question on the AI services you want to compare. Keep the question wording and testing conditions consistent across runs.

Repeating a question gives you more evidence than a single answer. A small sample does not establish a precise estimate of what the whole market sees.

Keep the full answer and its cited sources, alongside the question, service, surface, run number and timing. Record the model where it is shown and the relevant location and language settings. A finding should lead back to the answer that supports it.

3. Give each number a clear meaning

A mention rate answers a specific question: in what share of the answers reviewed in scope was the company named?

Mention rate = answers naming the company ÷ answers reviewed in scope.

State that scope beside the number: which buyer questions, services, surfaces and dates it covers. Show results by service as well as in total.

Keep the other outcomes separate:

  • Recommendation: Does the answer suggest the company as an option to choose? A name in a list is not enough.

  • Citation: Does the answer point to the company’s website as a source? State which answers support citation measurement.

  • Accuracy: Do claims about the company match confirmed facts? A citation alone does not establish that they do.

Use an appropriate, disclosed base for each measure. A question subset or a service’s source-reporting limits can change the relevant base.

If a measure was not assessed, label it as unmeasured. A blank field is not evidence of zero performance.

4. Make the retest comparable

Before publishing changes, define what the retest will hold constant: the question wording, service and surface, number of runs, collection rules and classification rules.

Record the publication dates of the work being evaluated. If a model changes, a service is added, or testing conditions differ, disclose it beside the comparison. Where possible, show the shared question-and-service set separately so readers can compare the same scope.

An improvement after publication is an observed change. It does not establish that the content caused it. Read the answers and the sources alongside the numbers before deciding what to do next.

5. Turn the findings into a marketing decision

A useful report should let your team answer three questions:

  • Where are we consistently absent on questions that matter to the business?

  • Where are we present but described incorrectly or given little reason to be chosen?

  • Which content or page change is worth making first, and what will we recheck afterward?

Those are different problems. They may require different work. A single blended score makes it harder to see which action the evidence supports.

Track the next stages separately, too. An AI mention, a website visit, a demo request and a customer are distinct outcomes. Better visibility is useful evidence; sales progress needs its own evidence.

How Signal Harbor puts this into practice

We connect the recorded buyer answers to a prioritized work plan. Your team can inspect the findings, review proposed content and page changes, and approve what gets published. A matched retest then shows what changed within the tested scope.

The goal is a report your marketing team can question, verify and use to decide where to spend its time.

Read our measurement methodology for the definitions and comparison rules.

If you want a starting point for your company, book a complimentary AI Visibility Snapshot and live portal demo. Bring the audience or product you want to focus on, and we will walk through the findings and the next useful step.

Book a Signal Harbor demo

Sebastian Miller
Co-Founder, Signal Harbor
signalharborai.com · LinkedIn

This article first appeared in Signal Harbor Weekly, the Signal Harbor newsletter on AI visibility.

All articles

Your next step

Find out where AI leaves you out.

See your AI Visibility Snapshot, explore the portal in a live demo, and discuss the next move for your team.

On the complimentary call

  • 1
    Your AI Visibility SnapshotHow AI answers describe and recommend your company today.
  • 2
    A live portal demoSee the audit, campaign, Content Studio, ads, and Results in action.
  • 3
    A clear next stepWhich plan fits, and where your first campaign would start.

The introductory call, AI Visibility Snapshot, and portal demo are complimentary. The Pro plan, Enterprise, and agency partnerships are paid plans. See what the Snapshot includes.