Large Language Models don't reason better than physicians - yet

Dr. Shantanu Nundy MD, MBA, is a Primary Care Physician and an FDA AI Advisor

Dr. Shantanu Nundy is back to unpack whether large language models truly reason better than physicians.

Every two weeks, Nundy dissects a study that’s making the rounds among his peers, breaking down the methodology and the paper’s impact.

His second review looks at the performance of a large language model on the reasoning tasks of a physician, published in Science on April 30, 2026. The full article is only available to subscribers behind a paywall.

If you have a study you’d like Dr. Nundy to tackle, please do reach out to us and we’ll ensure he gets your note!

🔥 HOT TAKE

A landmark study in Science has been touted as demonstrating that LLMs now reason better than physicians—but what it actually tested was differential diagnosis, not clinical reasoning.

Methodology: 9/10    
Importance: 6/10    
Likely impact: LLMs have come a long way, but we can’t yet declare that they are better than doctors at clinical reasoning.    
Ideal follow-up study: A turn-by-turn evaluation of how LLMs approach common diagnostic cases, assessing what questions they ask and how they synthesize extraneous information, including cases where the patient has no discernible diagnosis.

📋  CLIFF NOTES

Brodeur, Buckley, Kanjee, Goh et al. published what is arguably the most ambitious evaluation of LLM clinical performance to date in Science on April 30, 2026. The study team, spanning Harvard, Stanford, Beth Israel, and Microsoft Research, conducted five structured experiments comparing OpenAI’s o1 model against hundreds of physicians, then added a real-world arm with 79 randomly selected patients from the Beth Israel Deaconess emergency department in Boston.

The five experiments tested differential diagnosis on NEJM clinicopathological conference cases, NEJM Healer cases, triage differential, probabilistic reasoning, and management reasoning. In the real-world ED arm, o1 and two attending physicians gave blinded second opinions at three touchpoints: triage, initial evaluation, and admission. Evaluating physicians could not tell human from AI in over 83% of cases. In all experiments, o1 outperformed physicians.

🌐 BIGGER PICTURE

Let’s start with what this study got right because there is a lot. The authors tested across multiple case types, included hundreds of physician comparators, conducted real blinding with a formal check, and brought the evaluation into an actual emergency department with real patients. This is methodologically closer to how clinical AI should be evaluated than most of what has come before.

But the study’s central claim that it evaluated “reasoning tasks of a physician” deserves scrutiny: clinical reasoning, as Gruppen defines it in the Western Journal of Emergency Medicine, is fundamentally about the acquisition of information—the iterative process by which a physician decides what to ask, what to examine, what to order, and what not to. Basically, what every patient experiences when they go to see the doctor and the visit starts with the doctor asking a series of questions. In this study, the LLM never does any of that. Every case arrived after an expert human had already synthesized the history, chosen the exam findings, and decided which labs to include. The AI entered the loop after most of the cognitive work was done. It was generating differential diagnoses on a synthesized case, not using clinical reasoning to understand a real-world patient.

That LLMs now outperform physicians in differential diagnosis is a genuine technological achievement. Just a few years ago, it would have seemed implausible. But outperforming physicians on a pre-synthesized case is not the same as outperforming them on the full diagnostic process—and this paper never claimed to test the latter. The headlines did.

Reserve Your Spot for Upcoming Webinars!

💬 MY LONGER TAKE

A team from Curai Health captured the core limitation well in a piece published this month. They describe a scenario where a conversational AI takes a patient history, reasons fluently, and arrives at the correct diagnosis, but never asks about exertional onset, never elicits radiation to the jaw, and never establishes whether the chest tightness resolves with rest. On every current benchmark, it passes. In a real encounter, it could miss a STEMI. The problem isn't the reasoning; it's that the model doesn't know what it doesn't know. They call this the "ask vs act" gap: whether a system recognizes that critical information is still missing before it acts.

This matters because patients don’t present the way cases do in these studies. They don’t say “I’m a 45-year-old male with 20 pack-year smoking history, presenting with syncope.” They say, “I passed out” or “I don’t know how I got here. My family must have found me out cold and called the ambulance.” They also don’t always mention the night of heavy drinking. Knowing how to go from that to a structured clinical picture is the real clinical reasoning test.

Early efforts at simulating this information-gathering phase exist, but the evidence is mixed. Google’s AMIE (Tu et al., Nature, 2025), an LLM built specifically for diagnostic dialogue, matched or outperformed primary care physicians on text-based consults with patient actors. But the study authors acknowledged it was designed for how patients use LLMs, not how physicians practice medicine. On the other side, multi-turn evaluations like those by Sharma et al show the opposite signal: when LLMs must initiate questioning and adapt in real time, accuracy degrades compared to single-turn benchmarks.

Back to this paper: the primary metric was the Bond score, which measures whether the correct diagnosis appears in the differential (yes/no). That’s good but just as important as getting the right diagnosis is making sure you have considered and ruled out all the can’t-miss ones. The one place the study did that, on the NEJM Healer cases, o1-preview was not significantly better than physicians. The study also never estimated how harmful the LLM was when it was wrong, whether errors were clinically trivial or genuinely dangerous. Those are the most important questions from a regulatory perspective.

The best diagnosticians I’ve trained with were not just accurate—they were parsimonious. Their differentials were broad enough to catch the right answer, narrow enough to guide efficient workup, and disciplined enough to spare the patient unnecessary tests. A system that lists every ICD-10 code scores perfectly on accuracy and is clinically useless. Parsimony and diagnostic efficiency were not measured here.

The real-world ED arm is the most important part of the study—and it has a significant blind spot. Medicine’s hardest diagnostic challenge isn’t the admitted patient with something wrong. It’s the patient who has nothing, or nothing that warrants admission. In my own work on collective intelligence at Harvard, the case our physician network got most wrong was a patient with nausea whose diagnosis was simply nausea. No pathology. Just nausea. By filtering to admitted and ICU cases, this study excluded that population entirely. Most patients who come to an ED go home. An AI that handles admitted patients well but can’t recognize who is safe to discharge or has no real pathology is not ready for the full scope of emergency medicine.

For patients:

This paper makes a compelling case for one specific use of LLMs: getting a second opinion. Diagnostic errors affect roughly 1 in 20 outpatient visits. By pulling up their medical record through a patient portal and entering their doctor’s notes into an LLM, patients can cross-check a diagnosis and either gain confidence in their care plan or surface alternatives worth discussing with their physician. That’s a meaningful, low-risk application. It is also very different from the use case most people actually turn to LLMs for—symptom triage and self-diagnosis—a use case this paper was not designed to evaluate.

For AI developers and implementers:

The results support physician-facing clinical decision support. When information is pre-synthesized, LLMs are more than ready to guide differential diagnosis and management—as long as teams remain alert to the automation bias risks covered in the last Signal post. Patient-facing diagnostic AI is a different matter. We lack studies testing LLMs in the conversational, turn-by-turn environment where patients actually present—with incomplete histories, extraneous detail, and presenting complaints that may map to nothing. Deploying at scale in that context carries risks.

For policymakers and regulators:

This paper makes a strong case for an independent, real-world evaluation infrastructure. Multiple case types, large physician panels, real patients, formal blinding—this should be the floor, not the ceiling. But even here, admitted patients only, pre-synthesized presentations, and outcome metrics that miss most of what makes a good diagnostician. If we are serious about using AI to expand access to care, we need multi-dimensional evaluation frameworks built around the real and sometimes messy world of medicine.