The Model Isn’t the Medicine
Nature Medicine recently published a rigorous independent evaluation of clinical AI. Researchers at NYU compared three frontier large language models (GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6) against two purpose-built clinical AI tools, OpenEvidence and UpToDate Expert AI, across three evaluation stages: 500 standardized medical knowledge questions, 500 clinician-alignment items, and 100 real clinical queries (RCQs) drawn from physician use at NYU Langone Health, reviewed by twelve blinded clinicians.
The finding was decisive, and the response from the medical community was explosive. That’s because frontier models outperformed purpose-built clinical tools across every dimension. On real-world clinical queries, frontier LLMs scored roughly 10 to 14 percent higher in aggregate. The specialized tools performed comparably to auto-enabled Google Search.
I am an engineer who has spent my career building smart systems, not a doctor. From where I sit, independent, rigorous evaluation of clinical AI has been rare enough that its arrival warrants genuine attention. Health systems, procurement offices, and regulators should read this paper carefully. The evidence that frontier general-purpose models outperform purpose-built clinical tools on real physician queries is consequential.
`Upcoming Sessions
Live conversations with the people shaping healthcare.
Verifying AI in Medicine: Is Healthcare AI Actually Ready for Deployment?
Is AI ready for real-world clinical care? Dr. Zak Kohane and Protege’s Engy Ziedan debate what it takes to safely trust AI in diagnosis and treatment, from benchmark performance and clinical evidence to whether we’re holding AI to a higher standard than humans.
Why Are Hospitals Hiring Chief AI Officers?
As AI moves from pilots into day-to-day healthcare operations, health systems are having to decide who should actually own the
Where are AI Agents Actually Working in Healthcare, and Why?
While the healthcare industry remains locked in a high-level debate over the theoretical potential of artificial intelligence, a select group of
Top Employers Share What Healthcare Startups Need to Know in 2026
As enterprise health purchasers face volatile medical cost trends and tightening operational margins, the standard vendor pitch of "improved clinical efficiency&
Peptide Therapy: Where Do We Go From Here?
Watch the full webinar Peptide therapies have moved from niche biohacking circles into mainstream medicine, reshaping conversations around weight management, metabolic health,
Not everyone can access the top 1% of physicians. Will AI change that?
Historically, top-tier medical expertise was limited by wealth and geography. As AI reshapes healthcare delivery, a vital question emerges: can it democratize intelligence to give everyone access to the caliber of care typically reserved for the top 1%?
Privacy AI and the Future of HIPAA
Watch the Full Webinar As AI reshapes healthcare and more patient data flows beyond traditional clinical settings, the rules governing privacy, access,
What will AI do for employer healthcare and benefits?
Employer healthcare may be one of the most practical — and overlooked — proving grounds for AI.
Hospitals as the new go-to-market, lessons from the trenches
Watch the Full Webinar Health systems are entering a new frontier, one where innovation isn’t optional, and meaningful change requires creativity,