A medical policy analyst opens a criterion that was last reviewed before 2 new trials were published. Updating it means running a literature search, screening every abstract it returns, reading the studies that survive and writing a summary the committee can sign. Most of that time goes to the screening, and most of the abstracts screened will never matter. The criterion stays stale until someone finds the hours.
Coverage criteria are evidence syntheses with a deadline attached. Medicare Advantage makes the link explicit. Under 42 CFR § 422.101(b)(6), a plan that writes internal coverage criteria must base them on current evidence and publish a summary of the evidence it considered, a list of sources and the rationale for the criteria. That is the output of a review, whether or not anyone calls it one.
So the question health plans are starting to ask is a fair one. If AI tools can speed up systematic reviews, can they speed up medical policy? A new evidence map from the Agency for Healthcare Research and Quality gives the most careful answer we have seen, and it splits cleanly by task.
The tools are good at finding studies and weak at reading them
The AHRQ Evidence-based Practice Center program published its 2026 update of Machine Learning Tools To (Semi-)Automate Evidence Synthesis in August 2026 (AHRQ Publication No. 26-EHC011). It covers 168 studies, with searches running through April 4, 2026, and grades each review task by whether tools are ready for use.
| Review task | Median performance in the AHRQ map | AHRQ’s readiness call |
|---|---|---|
| Flagging randomized trials | 98% recall | Ready for use with oversight |
| Screening abstracts with active learning | 95% recall | Ready for use with oversight |
| Screening abstracts with an LLM alone | 87% recall | Not yet ready for routine use |
| Extracting descriptive data | 88% accuracy | Not yet ready for routine use |
| Extracting results | 54% accuracy | Not yet ready for use |
| Writing the search | 14% recall | Not yet ready for use |
The pattern matters more than any single number. The tasks that sort a pile of citations are close to dependable. The tasks that read a paper and pull out what it found are not. Active-learning screening, which reorders abstracts so the likely relevant ones come first, can cut screening work by about half while keeping recall high, according to the review.
For a coverage team, that split lands in a specific place. A criterion rarely turns on whether a trial exists. It turns on what the trial showed, in which patients, at what dose and against which comparator. That is results extraction, the task where the median tool got 54% right.
A criterion depends on what a trial found. That is the step where the tools are least reliable.
A missed trial is a coverage error
In academic reviews, recall is a quality metric. In medical policy it is closer to a safety metric. A screening step that drops 1 relevant trial can leave a criterion requiring a step therapy the newer evidence no longer supports, or excluding a population the trial included. The member who is denied under that criterion never sees the abstract that was screened out.
That makes the stopping rule the most consequential setting in an AI-assisted review. Active-learning tools rank abstracts, and someone has to decide when to stop reading down the list. The AHRQ review found that most heuristic stopping rules missed at least 1 relevant study in at least 1 simulation. One statistical rule, set to a 99% recall target, consistently found every study.
There is a second catch. Most of the studies in the map judged tools against the inclusion lists of published systematic reviews, which the authors call a noisy reference standard. A plan testing a tool on its own policies inherits the same problem. If the benchmark is last year’s evidence summary, the tool is graded on agreement with a human process that may itself have missed studies.
Test the tool again at every update
The review is blunt about how fast its findings age. Current models screen and extract better than earlier generations did, and the authors write that the pace of change limits the half-life of any specific finding. They also note that results depend heavily on prompts and that studies reported model versions and prompts inconsistently.
For a health plan, that rules out a one-time vendor assessment. Say a tool performed well on cardiology policies last spring. It may behave differently on a new model release or in a new clinical area. The practical answer is a small, repeatable validation set. Take a handful of past policy reviews where the plan knows which studies mattered, rerun the tool on each new release and record recall before using it on live work.
The same discipline belongs in the published record. Cochrane and 3 other evidence organizations have endorsed the RAISE recommendations, which call for disclosing the tool, the model and version, the prompts and the role of human oversight. A Medicare Advantage evidence summary already has to list its sources. Adding a short methods note on how those sources were found costs little. It tells a physician challenging the criterion, or an auditor, whether a screening model stood between the literature and the policy.
We have argued before that model documentation belongs in the coverage file for individual decisions. The same logic applies one level up. A criterion is a decision about thousands of future cases, and the evidence behind it should be as traceable as a single denial. Our case for publishing criteria the way engineers publish code adds the version history. A methods note completes it.
Where the human sits
None of this argues for keeping AI out of medical policy. Stale criteria are a real cost, borne by members and clinicians who are held to rules the evidence has moved past. Tools that cut screening time in half could let a policy team review criteria on a schedule set by the literature instead of the backlog.
The AHRQ authors put the bottom line plainly. No existing tool can replace human expertise, and the best ones keep humans in the loop. For coverage criteria, the loop has an obvious shape. The machine finds and ranks. A clinician reads the studies that matter, extracts what they showed and writes the rationale a plan has to defend.
Redlake builds tools for utilization review, and we hold our own work to the same split. A machine can say yes. Only a person can say no, and only a person should decide what the evidence says.