Checking an AI tool's work rarely stops because someone decides to stop it. It stops slowly, one right answer at a time, until nobody is doing it and nobody can say when it ended.
I want to show how that happens, and what goes missing with it.
An example I made up
Everything in this example is made up, including the numbers and the timings.
The situation
Chandra Mohan is head of accounts payable at a regional freight forwarder with its head office in Singapore. For eight months an AI tool called AnhingaAI has matched the team's supplier invoices to purchase orders and delivery records. It's Friday, and the week's batch is due to be paid that afternoon.
What the AI said
AnhingaAI's note on the batch said: "All 412 invoices in this batch match their purchase orders and delivery records. Amounts, supplier names and dates agree. I am 97 percent sure."
It was right about that. Every match in the batch was correct.
The check
When the tool was new, the team checked every match by hand. When a supplier's bank details changed, someone phoned the supplier on the number already on file before anything was paid. As the matches kept coming back right, the team began to check only a sample. Later they approved the batch as it came. The phone call stopped when the old check stopped.
That Friday, a new member of the team, two weeks into the job, asked a plain question. Three suppliers in the batch showed new bank details. Why had nobody phoned them?
The decision, and who was told
Chandra held those 3 payments. The team phoned each supplier on the number already on file. Two of the changes were real. The third came from an email the supplier had never sent.
Chandra approved 411 of the 412 invoices for payment, including the 2 with genuine new bank details. He stopped the 1 payment that rested on the false email. He made the phone call a named step, owned by the payments lead. He asked for 20 matches to be checked by hand every Friday. He told the finance director and the team what had happened. He also told the supplier whose name had been used.
Three months later
The phone call has stopped 1 more false request. The Friday sample is still in the team's calendar. The rule the team keeps is short: "When bank details change, we call the supplier on the number we already hold."
How checking stops
Nobody in Chandra's team decided to stop checking. In the first weeks they checked every match, because the tool was new. After enough right answers, a sample felt like enough. After enough good samples, approving the batch felt like enough. In the end approval was a click, and the checking had gone without anyone choosing to end it.
Each step made sense on the day it was taken. That's what makes it hard to see. In Chandra's team there was no meeting and no new rule. There was only a long run of answers that turned out to be right.
KPMG and the University of Melbourne surveyed more than 48,000 people in 47 countries between November 2024 and January 2025. Of the employees who use AI at work, 42% said they relied on AI output without evaluating its accuracy sometimes or more often. The survey doesn't say why. The example above shows one way people can come to rely on AI output without checking it.
What goes with it
This is the part that's easy to miss. When a team stops checking the AI's work, it can lose more than that check. It can also lose the checks that were never written down, because people made them without being asked, as part of looking at each invoice.
Phoning the supplier was never part of what AnhingaAI did. Nobody had asked it to confirm who owned a bank account. Its job was to match invoices to orders and deliveries, and it did that job well. The phone call happened because a person was looking at each invoice, and a person who looks at an invoice notices when the bank details have changed. When the looking stopped, the noticing stopped with it.
AnhingaAI's note was right about the matches. It said nothing about the bank accounts, because they were outside what it was asked to look at. A sure answer about one thing tells you nothing about the things the tool never looked at.
What a leader can do
Going back to checking everything by hand isn't the answer. The tool is doing useful work, and the team has other work to do. The aim is smaller and more exact than that.
Start by asking what the AI does not look at. Write that list with the people who did the work before the tool arrived, because they know which checks people made without being told to. In the example, the check that mattered most was one nobody thought of as a check.
Give each of those checks a name and an owner. In the example, the phone call became a step owned by the payments lead. If a named step is skipped, someone can see that it was skipped. A habit that nobody named can stop without anyone seeing it stop.
Keep a small sample checked by hand, on a fixed day. Chandra's team checks 20 matches every Friday. A sample done when there is time can fade into approving the batch, as it did before. A sample on a fixed day is easy to see when it is skipped.
Listen to the newest person in the team. In the example, the question came from someone two weeks into the job, who still saw the work the old way. When a new member of a team asks why something isn't done, it's worth finding out the reason before you answer.
What I won't claim
I made this example up, and its numbers are invented. I don't measure or score anyone's team, and this isn't a test of yours.
AnhingaAI can keep matching the invoices. A person in the team still has to decide which other checks stay, and make sure each one has someone who does it.
I give keynotes, briefings and workshops on this.