Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks
Specialized clinical AI assistants are rapidly entering medical practice, often framed as safer or more reliable than general-purpose large language models (LLMs). Yet, unlike frontier models, these clinical tools are rarely subjected to independent, quantitative evaluation, creating a critical evidence gap despite their growing influence on diagnosis, triage, and guideline interpretation. We assessed two widely deployed clinical AI systems (OpenEvidence and UpToDate Expert AI) against three state-of-the-art generalist LLMs (GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5) using a 1,000-item mini-benchmark combining MedQA (medical knowledge) and HealthBench (clinician-alignment) tasks. Generalist models consistently outperformed clinical tools, with GPT-5 achieving the highest scores, while OpenEvidence and UpToDate demonstrated deficits in completeness, communication quality, context awareness, and systems-based safety reasoning. These findings reveal that tools marketed for clinical decision support may often lag behind frontier LLMs, underscoring the urgent need for transparent, independent evaluation before deployment in patient-facing workflows.
692e607b37312eaa83fd87d9
Krithik Vishwanath
692e607b37312eaa83fd87da
Mrigayu Ghosh
692e607b37312eaa83fd87db
Anton Alyakin
claimed_verified
2025-12-03T09:18:26.406Z
66461ec9cce50c43b3d97e0b
/avatars/91f3175cdd1aad8e3192b6b37f490a07.svg
Anton
alyakin314
user
alyakin314
692e607b37312eaa83fd87dc
Daniel Alexander Alber
692e607b37312eaa83fd87dd
Yindalon Aphinyanaphongs
692e607b37312eaa83fd87de
Eric Karl Oermann
Generalist Large Language Models Outperform Clinical Tools on Medical Benchmarks
Specialized clinical AI assistants are rapidly entering medical practice, often framed as safer or more reliable than general-purpose large language models (LLMs). Yet, unlike frontier models, these clinical tools are rarely subjected to independent, quantitative evaluation, creating a critical evidence gap despite their growing influence on diagnosis, triage, and guideline interpretation. We assessed two widely deployed clinical AI systems (OpenEvidence and UpToDate Expert AI) against three state-of-the-art generalist LLMs (GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5) using a 1,000-item mini-benchmark combining MedQA (medical knowledge) and HealthBench (clinician-alignment) tasks. Generalist models consistently outperformed clinical tools, with GPT-5 achieving the highest scores, while OpenEvidence and UpToDate demonstrated deficits in completeness, communication quality, context awareness, and systems-based safety reasoning. These findings reveal that tools marketed for clinical decision support may often lag behind frontier LLMs, underscoring the urgent need for transparent, independent evaluation before deployment in patient-facing workflows.
Generalist LLMs outperform specialized clinical AI assistants in a multi-task benchmark, highlighting the need for independent evaluation of clinical decision support tools.
clinical AI assistants large language models LLMs OpenEvidence UpToDate Expert AI GPT-5 Gemini 3 Pro Claude Sonnet 4.5 MedQA HealthBench clinical decision support systems-based safety reasoning