A sepsis alert fires on a hospital ward. A request for a rehab stay reaches a health plan with a model’s recommendation already attached. Either way, a patient’s care now runs through software that the clinician acting on it usually cannot explain and that the patient may never see.
That alone does not rule the software out. Patients already take drugs whose mechanisms are still debated and trust tests they could never explain, and many would reasonably trade an understandable recommendation for a more accurate one. What a patient cannot be asked to accept is a decision nobody can account for. Black box AI earns its place in care only when the things around it hold up. The evidence has to fit the patient, and the human check has to be able to disagree. Above all, the decision has to be one that somebody can explain and the patient can challenge.
Accuracy has to be measured on the right patients
A headline accuracy figure tells a patient little until someone explains what it measured. Say a model recommending surgery has an excellent record of successful outcomes, and the patient is a pianist. Whether “success” includes the fine finger control her career depends on is the question that matters to her, and the figure does not answer it.
Developer figures may not match real-world performance either. Epic’s internal documentation put its sepsis model, then in use at hundreds of US hospitals, at an area under the curve of 0.76 to 0.83. When Michigan Medicine researchers tested it independently across 38,455 hospitalizations, they found 0.63, and at the hospital’s alert threshold it missed 67% of patients with sepsis. The authors said the gap was likely due to how the onset of sepsis was timed, and warned of a growing body of confidential, non-peer-reviewed vendor performance documents.
Population matters as much as outcome. When researchers tested leading dermatology models on a pathology-confirmed image set with diverse skin tones, the models showed substantial limitations, particularly on dark skin and uncommon diseases. A patient with dark skin who is quoted a model’s overall accuracy may be hearing a fact about other patients.
If you buy or deploy a clinical model, the questions are plain. Which outcome was measured, and in which patients? Who checked it outside the vendor?
A human check works only if it can disagree
The usual safeguard is a person between the model and the patient. That check is weaker than it looks. In a study published in Radiology, 27 radiologists read mammograms alongside AI suggestions that the researchers had made wrong for some cases. When the suggestion was wrong, inexperienced readers’ accuracy fell from almost 80% to under 20%. For readers averaging more than 15 years of experience, it fell from 82% to 45.5%.
The human alternative can fade as well. At 4 endoscopy centers in Poland, the share of colonoscopies done without AI in which doctors found a precancerous growth fell from 28.4% to 22.4% in the 3 months after the centers began using AI routinely, according to an observational study. A human option protects a patient’s choice only while it stays good enough to choose.
Design pulls the same way. Advice delivered through a warm, reassuring interface, with no real way to question it, can win agreement it has not earned, even from a patient who knows she is dealing with software. A human step built for agreement is not a check.
In coverage decisions, transparency has to stand in for consent
At a health plan, the patient is not in the room at all. Software may score a prior authorization request or suggest a determination before a nurse opens the case, and the member neither chooses it nor sees it.
The stakes show up in the numbers. A 2024 Senate subcommittee investigation found that UnitedHealthcare’s initial denial rate for post-acute care requests in Medicare Advantage rose from 8.7% in 2019 to 22.7% in 2022, as the company launched initiatives to automate prior authorization. Its overall denial rate barely moved, from 7.3% to 7.6%. The company said the report mischaracterized its clinical practices. Whatever the explanation, the arithmetic shows how an overall rate can hide where the denials land.
That is CMS’s position in its February 2024 guidance, which also restates that a physician or other appropriate professional must review adverse medical necessity decisions. A human decision is the right floor, and the mammography study shows why it is not enough on its own. When nobody asks the member, the decision has to carry its own explanation. The record should show which determination a model informed and which criteria and record facts it relied on. It should also name who made the call.
Where nobody is asked for consent, transparency has to do the work that consent would have done.
In our own tools, only a person can say no
We build tools that health plans use in utilization review, and Redlake AI, our lab, builds on knowledge graphs and domain ontologies. The members whose cases pass through those tools never chose them, so we hold ourselves to 3 rules.
- A clinician makes every denial. Our software may recommend and may approve, but every adverse determination is made by a licensed clinician. A machine can say yes. Only a person can say no.
- Every recommendation shows its reasons. Each cites the coverage criteria and the record facts it relied on, so the reviewer checks evidence rather than a score.
- Every decision can be reconstructed. We log recommendations and reviewer actions, so any determination can be rebuilt on appeal.
None of this opens the model itself, and none of it removes the pull toward agreement that the radiology study measured. A reviewer shown a tidy summary of evidence can still sign off without looking closely. The log at least makes that measurable. How often reviewers agree with the software, and on what evidence, becomes something a plan can check instead of something it has to assume.
The model can stay a black box. The decision cannot.