LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluation
The deployment of Large Language Models (LLMs) in high-stakes clinical settings demands rigorous and reliable evaluation. However, existing medical benchmarks remain static, suffering from two critical limitations: (1) data contamination, where test sets inadvertently leak into training corpora, leading to inflated performance estimates; and (2) temporal misalignment, failing to capture the rapid evolution of medical knowledge. Furthermore, current evaluation metrics for open-ended clinical reasoning often rely on either shallow lexical overlap (e.g., ROUGE) or subjective LLM-as-a-Judge scoring, both inadequate for verifying clinical correctness. To bridge these gaps, we introduce LiveMedBench, a continuously updated, contamination-free, and rubric-based benchmark that weekly harvests real-world clinical cases from online medical communities, ensuring strict temporal separation from model training data. We propose a Multi-Agent Clinical Curation Framework that filters raw data noise and validates clinical integrity against evidence-based medical principles. For evaluation, we develop an Automated Rubric-based Evaluation Framework that decomposes physician responses into granular, case-specific criteria, achieving substantially stronger alignment with expert physicians than LLM-as-a-Judge. To date, LiveMedBench comprises 2,756 real-world cases spanning 38 medical specialties and multiple languages, paired with 16,702 unique evaluation criteria. Extensive evaluation of 38 LLMs reveals that even the best-performing model achieves only 39.2%, and 84% of models exhibit performance degradation on post-cutoff cases, confirming pervasive data contamination risks. Error analysis further identifies contextual application-not factual knowledge-as the dominant bottleneck, with 35-48% of failures stemming from the inability to tailor medical knowledge to patient-specific constraints.
698e1234cace060ff123ab63
Zhiling Yan
claimed_verified
2026-02-13T09:37:17.425Z
65aae63fc3fa44c7109559bb
/avatars/b3f3e5d09b410f717c07b6aea997d595.svg
Zhiling Yan
JuelieYann
user
JuelieYann
698e1234cace060ff123ab64
Dingjie Song
claimed_verified
2026-02-13T09:37:19.177Z
619f01b8cc04eadf54fa5d5d
/avatars/928f3d1a6146e2e1ae4860445d929d5c.svg
Song Dingjie
songdj
user
songdj
698e1234cace060ff123ab65
Zhe Fang
698e1234cace060ff123ab66
Yisheng Ji
698e1234cace060ff123ab67
Xiang Li
698e1234cace060ff123ab68
Quanzheng Li
698e1234cace060ff123ab69
Lichao Sun
LiveMedBench: A Contamination-Free Medical Benchmark for LLMs with Automated Rubric Evaluation
The deployment of Large Language Models (LLMs) in high-stakes clinical settings demands rigorous and reliable evaluation. However, existing medical benchmarks remain static, suffering from two critical limitations: (1) data contamination, where test sets inadvertently leak into training corpora, leading to inflated performance estimates; and (2) temporal misalignment, failing to capture the rapid evolution of medical knowledge. Furthermore, current evaluation metrics for open-ended clinical reasoning often rely on either shallow lexical overlap (e.g., ROUGE) or subjective LLM-as-a-Judge scoring, both inadequate for verifying clinical correctness. To bridge these gaps, we introduce LiveMedBench, a continuously updated, contamination-free, and rubric-based benchmark that weekly harvests real-world clinical cases from online medical communities, ensuring strict temporal separation from model training data. We propose a Multi-Agent Clinical Curation Framework that filters raw data noise and validates clinical integrity against evidence-based medical principles. For evaluation, we develop an Automated Rubric-based Evaluation Framework that decomposes physician responses into granular, case-specific criteria, achieving substantially stronger alignment with expert physicians than LLM-as-a-Judge. To date, LiveMedBench comprises 2,756 real-world cases spanning 38 medical specialties and multiple languages, paired with 16,702 unique evaluation criteria. Extensive evaluation of 38 LLMs reveals that even the best-performing model achieves only 39.2%, and 84% of models exhibit performance degradation on post-cutoff cases, confirming pervasive data contamination risks. Error analysis further identifies contextual application-not factual knowledge-as the dominant bottleneck, with 35-48% of failures stemming from the inability to tailor medical knowledge to patient-specific constraints.
LiveMedBench addresses limitations in medical LLM evaluation by providing a continuously updated, contamination-free benchmark with rubric-based evaluation that better aligns with expert clinical reasoning.
Large Language Models medical benchmarks data contamination temporal misalignment clinical reasoning automated rubric-based evaluation multi-agent clinical curation framework