An example I made up
Caleb Tan is the deputy director of a public agency that renews business licences each year, in a city in Asia Pacific. The agency, the city, the timings and every number below are made up.
The situation
Most renewals change nothing from one year to the next. The agency's licensing assistant, RubythroatAI, proposes to decide those cases alone, so that officers no longer handle them. A few of them are refused, when the applicant says nothing has changed but the agency's records show something different. Caleb has a week to approve the proposal or reject it before the next renewal round opens.
What the AI proposed
Its recommendation reads: "Automate renewals and refusals for licences with no change since last year. The rules for these cases are clear. I am 93 percent sure."
The check
Caleb doesn't argue with the rules. He asks what a wrong decision would cost, for a renewal and for a refusal, and who would see it. A wrong renewal is found at the next inspection, and the agency can withdraw the licence. A wrong refusal stops a small business from trading. Its owner may not know how to ask for a review, so the agency may never hear about the mistake.
The decision
Caleb approves one part of the proposal. Renewals are automated. Every refusal the assistant suggests goes to an officer, who decides it within 2 working days. The head of licensing owns the rule, with a review in 6 months. Caleb tells the director and the officers. He also changes one line on the agency's public web page: "A person decides every refusal."
Three months later
Officers have overturned 11 of the 40 refusals the assistant suggested. Most were cases where the form said no change, the records showed a new address, and the business had only moved within the same district. Under the original proposal, each of those 11 refusals would have gone out with no person looking at it.
The proposal grouped decisions by how clear the rule was. What mattered was whether a wrong decision could be seen and undone.
My test before an AI decides alone
I'd say a decision is right to automate when it passes four tests.
A wrong decision will be seen
Someone at the organisation will notice it without waiting for the person who was harmed to complain. A wrong renewal is seen when an inspector visits. A wrong refusal may never be seen by anyone.
It can be undone at a cost the organisation accepts
Undoing it has to be possible, and the cost of undoing it has to be one the organisation has agreed to carry. A wrong renewal can be found and withdrawn at the next inspection. A business refused by mistake may close before anyone looks again.
One named person owns the rule and the date to look at it again
A rule with no owner keeps running after the facts around it have changed. Put the owner's name and the review date in the decision itself.
The person affected can ask a person
If the decision goes against them, they can reach someone who is able to look at it and change it. They also need to know that they can.
If one of the four tests fails, I'd keep that part of the decision with a person. Often that means splitting the group, as Caleb did. Notice that a clear rule isn't on the list. Clear rules help, but they don't tell you what happens when the AI applies them to the wrong case.
Why a person beside every AI decision isn't enough by itself
One answer is to put a person beside the AI and have them approve each decision. I don't think that's enough by itself.
A pre-registered meta-analysis by Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone (2024) pooled 106 experiments from 74 papers, covering January 2020 to June 2023. Across those experiments, people with AI beat people alone. But a person and an AI together did worse than the better of the two working alone, most of all on decision tasks.
Being beside the AI is also not the same as checking it. In their 2025 study, KPMG and the University of Melbourne surveyed more than 48,000 people in 47 countries. Of the employees who use AI at work, 42% said they relied on AI output without evaluating its accuracy. They said this happened sometimes or more often.
So the person beside the AI has a job to do, and it has to be possible to do it. They need to:
- open the case itself, and not only the AI's summary of it;
- look for what would show the answer is wrong, such as whether a business had left its district or had only moved within it;
- have the authority to overturn the AI's answer, and the time to do it;
- record each overturn and the reason, so the owner of the rule sees the pattern.
Eleven overturns out of 40 is information about the rule. That count should reach the head of licensing before the 6-month review, because it may be a reason to change the rule early.
If the person can't do these things, they only sign the AI's answer. The record will still say a person decided.
What I won't claim
The study above compares averages across many different experiments. It doesn't show which decisions to automate, and it doesn't show that an AI alone decides better than people do. Its experiments differ in task and set-up. It doesn't show how a person who approves each AI decision would do in a licensing office.
The 42% is what people said about their own habits. It isn't a count of how often they were wrong.
My four tests are my own view, not a published standard, and a proposal that meets them isn't proven safe. They're what I'd want to see met before I approved a proposal like the one Caleb received.
The example is made up. It isn't drawn from any real agency or licensing scheme.
If your leaders are considering a proposal like this
The four tests are a good way to start the conversation in the meeting where the proposal is approved. I give keynotes, briefings and workshops on this.