A Human in the Workflow Is Not a Human in the Loop

A care coordinator at a health insurer reviews post-acute claims for elderly patients. An algorithm has already predicted how many days each patient should need in a skilled nursing facility. The algorithm says 17 days. The patient's physician says 25. She has a screen full of cases and a target: keep actual stays within 1% of what the model predicts. She clicks approve. Next case.

That is not a thought experiment. UnitedHealth deployed a system called nH Predict to manage rehabilitation coverage for Medicare Advantage patients, and a STAT investigation found employees were pressured to follow its length-of-stay predictions. When patients appealed, 90% of denials were reversed. A human was in the loop for every one of those decisions. The outcomes suggest the human changed almost nothing.

The same shape shows up without any AI at all. ProPublica found Cigna physicians rejecting more than 300,000 flagged claims in two months, at an average of 1.2 seconds per claim, without opening a patient file. The tool was a simple matching system. The failure was the review.

Approved is not reviewed

The three kinds of work names this exactly. There is work that got cheap, work that is still yours, and a third kind: catching what the machine got wrong. The third kind is thin per instance and it accumulates. Nobody budgets for it, so it quietly becomes clicking approve.

The research on why is uncomfortable. In a controlled experiment on judicial decisions, people who assessed a case first and then saw incorrect AI advice got it right 66% of the time. People who saw the wrong answer first got it right 37% of the time. Showing people the answer before they think made them worse than having no help at all.

And the better the system usually is, the worse this gets. A system that is right 95% of the time trains its reviewers to stop questioning it. The 5% sails through.

Why the review decays

Three things break it, and none of them is discipline. Volume: if thoughtful review takes three minutes and the queue has 200 items, the reviewer adapts by doing less. Reliability: errors are rare, so the reviewer stops looking. Incentives: if you disagree and you are wrong, it is visible; if you agree and the system is wrong, it is the system's fault.

For an engineer this is the same failure as reading twenty-six lines that looked fine. On the delegation ladder, review is the checkpoint that justifies a rung. If the checkpoint is a rubber stamp, the work is running a rung higher than anyone agreed to, and nobody knows it.

What a real check needs

A real control detects problems, allows timely intervention, and can be measured. So give the reviewer a specific mandate: what to check, against what, with the authority to override. Let them form a judgment before they see the recommendation. Review a fraction thoroughly rather than everything superficially. Track the override rate; if it is near zero, either the model is flawless or nobody is reviewing. Insert cases where the answer is deliberately wrong and see whether they are caught.

If the mandate is vague, the override rate is unmeasured, and the review time looks like confirmation, you do not have a human in the loop. You have a human in the workflow. Those are not the same thing.

Adapted from Human in the Loop Is Not an AI Control in The Forward View. Read the full issue.