An example I made up
Everything in this example is made up, including the numbers.
The situation. Harith Zainal is head of customer operations at a general insurer in Kuala Lumpur. Six months ago he set his teams a target: the AI assistant used on 70% of customer cases. Each month he reports on it to the leadership meeting. This month's meeting is the next morning.
What the AI produced. He asks ParrotfinchAI to draft his monthly summary. Its line reads: "AI use reached 81% of cases in September, above the 70% target. Based on the assistant's activity log for the month. I am 97 percent sure." The count is right. The log shows exactly what the summary says.
The check. Before the meeting, Harith reads ten cases from the team whose use was highest. In two of them, the officer on the case used the assistant to answer a hardship request. That is a customer asking for more time to pay. The assistant wasn't set up for that kind of request. A team lead had to phone both customers again to correct the answer.
The decision. Harith removes the use target from the next month. He tells his team leads why, in person, before the leadership meeting. At the meeting he still reports the 81%. He adds one sentence on why use is no longer a target. It was pushing people to use the assistant on cases it wasn't set up for.
What happened after. Three months later, use had settled near 60% of cases. The repeat phone calls on hardship requests had stopped.
What a use count tells you
A use count tells you how often people opened the tool. That's worth knowing. If almost nobody has opened it six months after launch, you want to find out why.
But that is all the count holds. In Harith's example the log was accurate. It recorded every case where the assistant was opened, and the summary reported it faithfully. Whether the assistant's answer fitted the case is a different question, and the log holds nothing on it.
I think this is why a use count is so easy to read as progress. The number goes up and passes the target someone set. On a slide, beside other rising numbers, it looks like things are getting better. With AI use, a higher number only means more cases went through the tool. It says nothing about whether those cases were handled well.
What happens when use becomes the target
This is the part I'd ask any leader to think hard about. When use is the target, I expect people to reach it.
The people in Harith's teams were doing what the target asked of them. People first use the assistant on the cases it fits well. Then the target pushes them to use it on cases it fits worse. One was the hardship request, which needed a person to decide what the insurer could offer.
The cost of those cases doesn't appear in the use count. It appears later and somewhere else, in a team lead phoning a customer for the second time. Meanwhile the count keeps rising. So the report looks better at the same moment the work on those cases gets worse.
Look at what Harith actually decided. He kept the count and reported it to the leadership meeting. He removed only the target, because the target was what pushed the tool onto cases it didn't fit. And he told his team leads first, in person, because they were the people who would change how their teams worked. The leadership meeting heard the same reason, in one sentence.
When use settled near 60%, I'd still read that as progress, because the repeat calls on hardship requests had stopped. What mattered in his example was in two of the ten cases, and he saw it only because he read them.
What I'd look at instead
I'd look at the decisions. Each month, read about ten cases the way Harith did. Ask what the person decided with the AI's answer, and how they checked it. That tells you whether the tool is being used where it fits, and whether your people are still checking what it gives them.
ParrotfinchAI drafted an accurate summary. Whether the target behind that summary was still the right target was Harith's decision. AI can draft the plan. It cannot decide whether the plan is right.
I won't claim to measure or score anyone's use of AI, or to rank teams by it. The example above is made up.
I give keynotes, briefings and workshops on this.