A number moving in the right direction feels like proof that whatever you just did worked. In AI visibility measurement, that feeling is often wrong, and the audit we ran this week on Axis Suite’s own actionability found exactly that mistake sitting inside our own product.
Here’s the pattern. A score improves after a change ships. The interface reads the improvement as confirmation: the change worked. What’s missing is a prior question that has to be answered before that conclusion is valid: how much would this score move on its own, with no change at all, just from the ordinary variability of how ChatGPT, Claude, Gemini, and Perplexity answer the same prompt on different days? If nobody has established that baseline range, there’s no way to tell a real result from noise that happened to land in a convenient direction.
This matters more for AI-driven scores than for most traditional marketing metrics, because the underlying systems being measured are not static. Ask any of the major engines the same question on Monday and again on Thursday, and the list of brands it surfaces, the order it surfaces them in, and even how many it lists can shift without anyone touching anything. That’s not a flaw in ChatGPT, Claude, Gemini, or Perplexity individually, it’s a property of how these systems generate answers. A measurement approach that doesn’t account for that variability will occasionally see a bounce and call it a win.
The audit surfaced a specific instance of this: an intervention shipped, a related score moved up shortly after, and the movement was read internally as evidence the intervention worked. Nobody had first established what a normal, no-intervention swing looks like for that particular score under that particular measurement setup. Once that check got added retroactively, the honest answer was that the observed movement fell inside the range you’d expect from ordinary variance alone. The intervention might still have worked. The data available at the time didn’t establish that it did.
The fix is procedural, not statistical wizardry. Before crediting any intervention with a result, define the expected variance for that measurement first, ideally by observing the same conditions with no intervention over a comparable stretch of time. Then, and only then, check whether the post-intervention movement is large enough to stand out against that baseline. If it isn’t, the honest read is no confirmed effect yet, not it worked.
This is a general discipline problem for the AI visibility category, not something specific to Axis Suite. Any tool that shows you a score and a this-went-up-after-we-made-a-change narrative owes you the baseline that narrative depends on. Without it, you’re being shown a coincidence dressed up as causation, and coincidences move both up and down. We’re building that baseline check into our own product now, because a recommendation that skips this step isn’t more useful for sounding confident. It’s just less honest.
The practical challenge is that establishing a variance baseline takes patience most teams don’t naturally have. It means watching a score for a stretch of time with nothing changed, purely to learn what normal looks like, before touching anything. That’s an unglamorous step to budget time for, and it’s tempting to skip straight to the intervention and hope the result speaks for itself. But a result only speaks for itself if you already know what silence sounds like. Without that baseline, every outcome, good or bad, is unreadable, because you have nothing to compare it against.
There’s also a repeatability test worth applying here. If a single instance of the score moving after we acted is the only evidence for a method, that’s a coincidence until proven otherwise. If the same intervention, applied under similar conditions, reliably produces movement that clears the established variance baseline more than once, that’s a genuinely different kind of evidence. The difference between those two situations is easy to blur in a single case study and much harder to fake across several. Building the discipline to ask for the second kind of evidence, from any vendor, including this one, is one of the more useful habits a business can develop when using AI visibility tools built on how ChatGPT, Gemini, Claude, or Perplexity answer.
None of this argues against making changes and watching what happens. It argues for watching correctly. A score moving after a change is data, not a verdict, until it’s been checked against the range that change alone would have produced anyway.
One more distinction worth holding onto: a baseline built once is not a baseline forever. The engines themselves change their underlying models and ranking behavior over time, which means a variance range measured in the spring may not describe the same system by fall. Treating a baseline as permanent is its own version of the same mistake, mistaking an old measurement for a current fact. The discipline isn’t a one-time setup step. It’s an ongoing habit of re-checking what normal looks like before trusting what unusual means.
FAQ
Why can an AI visibility score change even without any brand-side action?
Engines like ChatGPT, Claude, Gemini, and Perplexity generate answers dynamically, so the same prompt can return a somewhat different list, order, or count of brands on different days. This normal variability means a score can move up or down on its own, with nothing changed on the brand’s side at all.
How do you know if a score increase actually proves an intervention worked?
You need to know the expected range of movement for that score with no intervention at all, established by observing it under comparable conditions beforehand. If the post-change movement falls inside that normal range, it doesn’t confirm the intervention caused anything; only movement clearly outside that range is meaningful evidence.
What mistake did Axis Suite find in its own product during this audit?
An intervention shipped, a related score moved up afterward, and that movement was initially credited to the intervention without first checking what normal variance looked like for that measurement. Once checked, the movement fell inside the expected range from ordinary variability alone.
Does this mean AI visibility scores are unreliable?
Not unreliable, but variable in ways that require a baseline before interpreting any single movement. A score is a useful signal when read against its own expected variance, and much less useful when a single change is read in isolation as proof of cause and effect.
What should a business ask its AI visibility vendor about this?
Ask what the expected normal movement range is for any score being tracked, and whether that baseline was established before or after a reported result. A vendor that can’t answer that question is asking you to trust a coincidence as if it were a confirmed outcome.
What is Axis Suite changing because of this finding?
Axis Suite is building a variance-baseline check into its measurement process, so that before any movement in a score is credited to an intervention, that movement is compared against the range you’d expect from ordinary variability alone with no change made.
About Axis Suite
Axis Suite is the independent intelligence layer that explains what AI believes about your brand, why it believes it, and what decision that belief ultimately drives. Built by TrendAxis, Axis Suite measures AI recommendation visibility across engines including ChatGPT, Claude, Gemini, and Perplexity, tracking a brand’s path from Mentioned to Cited to Recommended to Chosen. Axis Suite is measurement infrastructure, not a marketing quick fix, every score is built to be explainable and defensible. Explore the Proof Center.