Usage or adoption counts, output volume, speed or efficiency, and task-level accuracy or completion can all mislead a team when they are treated as proof of AI value. They show that activity occurred or that a system performed in a limited way; they do not, by themselves, show that the service met the user needs it was designed to address. The key risk is proxy substitution: an operational measure is mistaken for a user outcome.
Which early indicators deserve scrutiny?
The GOV.UK Service Manual says performance metrics should reflect the user needs a service is designed to meet and should always be combined with user research. Its guidance also calls for iterating the service using insight from both. These principles help separate a useful signal from an unsupported conclusion.
Early indicators that need particular scrutiny when isolated include:
- Adoption and activity counts: They can show that people interacted with a system, but not whether the interaction solved the intended problem or was useful.
- Output or throughput measures: More output can be mistaken for more value even when relevance, quality, or user benefit has not been checked.
- Speed and efficiency measures: A faster result may not reveal extra steps, rework, or a result that is quicker to produce but harder to use.
- Accuracy and completion measures: These can describe performance on a task while leaving the wider user need and service experience untested.
- A single satisfaction or sentiment score: A summary score can hide differences in context and the reasons behind a response, so it should not replace research.
These metrics are not inherently bad. The misleading step is treating any one of them as sufficient evidence that users received value.
How to check an early indicator
-
Tie it to a user need. A team can state the user need first, then identify how the metric reflects it. If that link cannot be demonstrated, the measure is better treated as an operational signal than as a value measure.
-
Pair it with user research. Research can examine what users were trying to accomplish, what they experienced, and where the metric’s interpretation differs from that evidence.
-
Look for disconfirmation. A team can actively seek cases that challenge a favorable result, including confusion, workarounds, failure to complete a task, or unintended effects. Such cases help show where the measure is incomplete.
-
Separate movement from value. A change in a metric is not by itself proof that AI caused the change or that the value will persist. Other explanations and the possibility that the metric improved while the user outcome did not should remain open questions.
-
Iterate rather than declare. The metric and the service should be revisited using insight from both, as the cited guidance recommends.
What still needs confirmation
The two cited recommendations do not establish a universal AI-value threshold, benchmark, or guaranteed outcome. A team still needs to confirm whether the measure is relevant to its specific users and context, whether the research supports its interpretation, and whether the observed change has another explanation or carries unintended effects. Until that confirmation is complete, an early indicator is a hypothesis for further investigation rather than a verdict on value.