Review: Our first in a new series evaluating the latest medical AI studies

Dr. Shantanu Nundy MD, MBA, is a Primary Care Physician and an FDA AI Advisor

Readers, we’re always looking to evolve in how we cover the intersection of health and technology. So we couldn’t say no when AI expert Dr. Shantanu Nundy offered to review the latest research globally, and unpack the relative importance. There’s an immense amount of noise in the market today and not enough signal.

So every two weeks, he will unpack a study that’s making the rounds amongst his peer set and dig into the methodology and likely impact.

If you have a study you’d like Dr. Nundy to tackle, please do reach out to us and we’ll ensure he gets your note!

His first review looks at a study with the title: Automation Bias in LLM-Assisted Diagnostic Reasoning Among AI-Trained Physicians, which was published on April 23, 2026. The full article is only available to subscribers behind a paywall.

🔥 HOT TAKE

A new RCT in NEJM AI shows that even doctors trained to use AI are biased by bad AI recommendations. This suggests the problem isn't with physicians; it's that we have no independent way to know when AI is wrong in the first place.

Methodology: 8/10
Importance: 10/10
Likely impact of the study: Should make health systems more wary about doctor-facing AI without rigorous independent benchmarking
Ideal follow-up study: Replicating study in US context with specialty-matched cases and doctors

📋  CLIFF NOTES

Qazi et al. conducted a single-blind randomized clinical trial enrolling 44 physicians across multiple institutions in Pakistan, all of whom had completed a rigorous 20-hour AI literacy training covering LLM capabilities, prompt engineering, and critical evaluation of AI outputs. Physicians were randomized 1:1 to diagnose six clinical vignettes with either accurate ChatGPT-4o suggestions (control) or suggestions containing deliberately introduced errors in 3 of 6 cases (treatment).

Importantly, LLM consultation was entirely voluntary. Physicians could choose to consult, modify, or ignore AI output at any point. Despite their training, physicians exposed to erroneous AI showed a 14 percentage-point drop in diagnostic accuracy.

To me, more interesting than the headline result are two subgroup analyses: 1) more experienced physicians who at baseline had higher diagnostic accuracy had a greater degradation in their diagnostic performance than less experienced physicians (statistically significant); 2) physicians who reported more frequent LLM use prior to the study tended towards greater degradation in diagnostic performance (not statistically significant). These results, if confirmed in other studies, would suggest that as physicians use these tools more, we may paradoxically see worse outcomes, not better.

Reserve Your Spot for Upcoming Webinars!

🌐 BIGGER PICTURE

This finding doesn't exist in isolation. A growing body of evidence — from Goh et al. in JAMA Network Open to the "From Tool to Teammate" RCT published last month in npj Digital Medicine — consistently shows that how AI is deployed matters as much as its performance in isolation. But this paper adds a darker dimension: you can train physicians on how to use AI, give them autonomy on when and how to use it, and it still may not be enough.

So what does that mean for healthcare leaders and investors? The U.S. clinical AI market is projected to exceed $45 billion by 2030, yet much of the investment and attention is still going to building clinical AI systems and demonstrating their performance on curated benchmarks. What it hasn't built is the infrastructure to evaluate those systems in conditions that reflect clinical reality: adversarial inputs, diverse patient populations, and real physician workflows. That’s the gap the field needs to close.

💬 MY LONGER TAKE

I've spent the better part of two decades at the intersection of human and machine intelligence in medicine. The finding that jumps out to me isn't the 14-point accuracy drop. It's that it happened in physicians who chose to consult the AI. These were engaged, trained clinicians exercising judgment about when AI was appropriate to use, and even then, the AI misled them. That tells me the problem is upstream of the physician entirely.

Study design

Strength: physicians were scored on the quality of their diagnostic reasoning, not just their final diagnosis, and both measures of performance showed significant drops in the treatment group. This matters because reasoning through a case is more representative of practicing medicine in the real world than a yes/no diagnosis based on a fully constructed case. But reasoning is harder and more costly to measure, and so is often not done in these studies.

Weakness: the vignettes were not matched to physician specialty, meaning surgeons were tested on internal medicine cases and vice versa. That's unfortunately common in these studies and in widely used tools like Open AI’s Healthbench, but highly problematic. Give an internist who hasn't seen a pediatric patient in 20 years a peds case and you're likely to get more automation bias. That alone may partially explain why more experienced physicians, who are more specialized and further out from general medical training, were more subject to automation bias.

Implications

Policymakers: We don't just need benchmarks that measure the accuracy of clinical LLMs in isolation. We need benchmarks that measure how AI affects physicians and patients in simulated or real clinical environments. That requires independent infrastructure that doesn't currently exist: not built by vendors, or by academia, but constructed by people who understand both the science and the clinical reality.

AI developers: Before any clinical AI tool reaches a physician, it should be stress-tested with inputs designed to make it fail: not the clean, well-formed cases it was trained on, but the ambiguous, atypical presentations that constitute the real diagnostic challenge. Evaluation also shouldn't stop at the final answer. It should assess the reasoning chain, because as this paper shows, it's the AI's explanation that shapes how clinicians think, not just the output they accept or reject.

AI implementers: Clinical AI systems should be required to display their uncertainty explicitly, so physicians can gauge the strength of a recommendation, the way clinical guidelines grade evidence. Every AI-assisted clinical decision should generate an audit trail to enable the kind of retrospective learning that could actually measure, at scale, automation bias and impact on physician decision making.

About the Author - Dr. Shantanu Nundy

Dr. Shantanu Nundy, MD, is a serial innovator, practicing physician, and advisor on artificial intelligence to the FDA in the Commissioner’s Office. Dr. Nundy has spent the past twenty years bringing innovative tools and emerging technologies to drug development, clinical practice, and public health. He pioneered the use of collective intelligence (humans + AI) in medicine and served as Chief Health Officer at Accolade, an AI-enabled navigation platform serving over 10 million Americans.

He is the author of two acclaimed books, Stay Healthy At Every Age and Care After Covid, and practices primary care part-time in the safety net. He holds a BS from MIT, an MD from The Johns Hopkins University School of Medicine, and an MBA from the University of Chicago.