Generalist Foundation Models Are Not Clinical Enough for Hospital Operations
Hospitals and healthcare systems rely on operational decisions that determine patient flow, cost, and quality of care. Despite strong performance on medical knowledge and conversational benchmarks, foundation models trained on general text may lack the specialized knowledge required for these operational decisions. We introduce Lang1, a family of models (100M-7B parameters) pretrained on a specialized corpus blending 80B clinical tokens from NYU Langone Health's EHRs and 627B tokens from the internet. To rigorously evaluate Lang1 in real-world settings, we developed the REalistic Medical Evaluation (ReMedE), a benchmark derived from 668,331 EHR notes that evaluates five critical tasks: 30-day readmission prediction, 30-day mortality prediction, length of stay, comorbidity coding, and predicting insurance claims denial. In zero-shot settings, both general-purpose and specialized models underperform on four of five tasks (36.6%-71.7% AUROC), with mortality prediction being an exception. After finetuning, Lang1-1B outperforms finetuned generalist models up to 70x larger and zero-shot models up to 671x larger, improving AUROC by 3.64%-6.75% and 1.66%-23.66% respectively. We also observed cross-task scaling with joint finetuning on multiple tasks leading to improvement on other tasks. Lang1-1B effectively transfers to out-of-distribution settings, including other clinical tasks and an external health system. Our findings suggest that predictive capabilities for hospital operations require explicit supervised finetuning, and that this finetuning process is made more efficient by in-domain pretraining on EHR. Our findings support the emerging view that specialized LLMs can compete with generalist models in specialized tasks, and show that effective healthcare systems AI requires the combination of in-domain pretraining, supervised finetuning, and real-world evaluation beyond proxy benchmarks.
692066f38c38b39d6a482dbd
Lavender Y. Jiang
claimed_verified
2025-11-21T16:25:09.686Z
6143be23a05498d58df55450
https://cdn-avatars.huggingface.co/v1/production/uploads/6143be23a05498d58df55450/IexeJ6WaNFuFX4-b3Gxv4.jpeg
Lavender Jiang
LavenderJ
user
LavenderJ
692066f38c38b39d6a482dbe
Angelica Chen
claimed_verified
2025-11-21T16:25:04.382Z
6296171e91760de5e5482894
/avatars/f6209e822f78933f4ee6869f7f4dc704.svg
Angelica Chen
angie-chen55
user
angie-chen55
692066f38c38b39d6a482dbf
Xu Han
claimed_verified
2025-11-24T07:58:23.138Z
66424b63273e548725934f7c
https://cdn-avatars.huggingface.co/v1/production/uploads/66424b63273e548725934f7c/P85o1t1WwYSAzYbBhW0GM.jpeg
Xu Han
xh852
user
xh852
692066f38c38b39d6a482dc0
Xujin Chris Liu
claimed_verified
2025-11-24T07:58:25.009Z
60920908c4a92d63a91c522a
/avatars/fa6c319913b26b9f177ea3ce9fe71aae.svg
Chris Liu
Aceticia
user
Aceticia
692066f38c38b39d6a482dc1
Radhika Dua
claimed_verified
2025-11-21T16:49:01.926Z
67649cd08989871cde263811
/avatars/57eb503ddde82b1e44f3795781c2f954.svg
Radhika Dua
Radhikadua123
user
Radhikadua123
692066f38c38b39d6a482dc2
Kevin Eaton
692066f38c38b39d6a482dc3
Frederick Wolff
692066f38c38b39d6a482dc4
Robert Steele
claimed_verified
2025-11-21T16:49:03.972Z
67c7f2a3252114c63bfea097
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/75MAkb33OFBXNwkFWIY69.png
Robert Steele
re-bort
user
re-bort
692066f38c38b39d6a482dc5
Jeff Zhang
692066f38c38b39d6a482dc6
Anton Alyakin
claimed_verified
2025-11-30T20:10:32.865Z
66461ec9cce50c43b3d97e0b
/avatars/91f3175cdd1aad8e3192b6b37f490a07.svg
Anton
alyakin314
user
alyakin314
692066f38c38b39d6a482dc7
Qingkai Pan
claimed_verified
2025-12-08T08:36:02.031Z
6933330e2bcd9e6ff0d569d2
/avatars/9bd327b1133962638fda2e5e4a555958.svg
Qingkai Pan
ivrics
user
ivrics
692066f38c38b39d6a482dc8
Yanbing Chen
692066f38c38b39d6a482dc9
Karl L. Sangwon
claimed_verified
2025-11-24T07:58:26.753Z
63697800bf21b20c568f1103
/avatars/cd57cf0d5596ff2df4155b26c65ae879.svg
Karl L. Sangwon
KarlSangwon
user
KarlSangwon
692066f38c38b39d6a482dca
Daniel A. Alber
claimed_verified
2025-11-21T16:25:06.313Z
644a848244b75fd958062ba3
https://cdn-avatars.huggingface.co/v1/production/uploads/644a848244b75fd958062ba3/d7vrsMSnVaBmpw3TfBVGO.jpeg
Daniel Alber
dalber
user
dalber
692066f38c38b39d6a482dcb
Jaden Stryker
692066f38c38b39d6a482dcc
Jin Vivian Lee
692066f38c38b39d6a482dcd
Yindalon Aphinyanaphongs
692066f38c38b39d6a482dce
Kyunghyun Cho
692066f38c38b39d6a482dcf
Eric Karl Oermann
Generalist Foundation Models Are Not Clinical Enough for Hospital Operations
Hospitals and healthcare systems rely on operational decisions that determine patient flow, cost, and quality of care. Despite strong performance on medical knowledge and conversational benchmarks, foundation models trained on general text may lack the specialized knowledge required for these operational decisions. We introduce Lang1, a family of models (100M-7B parameters) pretrained on a specialized corpus blending 80B clinical tokens from NYU Langone Health's EHRs and 627B tokens from the internet. To rigorously evaluate Lang1 in real-world settings, we developed the REalistic Medical Evaluation (ReMedE), a benchmark derived from 668,331 EHR notes that evaluates five critical tasks: 30-day readmission prediction, 30-day mortality prediction, length of stay, comorbidity coding, and predicting insurance claims denial. In zero-shot settings, both general-purpose and specialized models underperform on four of five tasks (36.6%-71.7% AUROC), with mortality prediction being an exception. After finetuning, Lang1-1B outperforms finetuned generalist models up to 70x larger and zero-shot models up to 671x larger, improving AUROC by 3.64%-6.75% and 1.66%-23.66% respectively. We also observed cross-task scaling with joint finetuning on multiple tasks leading to improvement on other tasks. Lang1-1B effectively transfers to out-of-distribution settings, including other clinical tasks and an external health system. Our findings suggest that predictive capabilities for hospital operations require explicit supervised finetuning, and that this finetuning process is made more efficient by in-domain pretraining on EHR. Our findings support the emerging view that specialized LLMs can compete with generalist models in specialized tasks, and show that effective healthcare systems AI requires the combination of in-domain pretraining, supervised finetuning, and real-world evaluation beyond proxy benchmarks.
Lang1, a specialized language model pretrained on clinical data, outperforms generalist models in predicting hospital operational metrics through supervised finetuning and real-world evaluation.
foundation models Lang1 pretrained models clinical tokens EHRs REalistic Medical Evaluation ReMedE 30-day readmission prediction 30-day mortality prediction length of stay comorbidity coding insurance claims denial AUROC zero-shot settings finetuning cross-task scaling in-domain pretraining specialized LLMs supervised finetuning real-world evaluation