CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcare
Integrating language models (LMs) in healthcare systems holds great promise for improving medical workflows and decision-making. However, a critical barrier to their real-world adoption is the lack of reliable evaluation of their trustworthiness, especially in multilingual healthcare settings. Existing LMs are predominantly trained in high-resource languages, making them ill-equipped to handle the complexity and diversity of healthcare queries in mid- and low-resource languages, posing significant challenges for deploying them in global healthcare contexts where linguistic diversity is key. In this work, we present CLINIC, a Comprehensive Multilingual Benchmark to evaluate the trustworthiness of language models in healthcare. CLINIC systematically benchmarks LMs across five key dimensions of trustworthiness: truthfulness, fairness, safety, robustness, and privacy, operationalized through 18 diverse tasks, spanning 15 languages (covering all the major continents), and encompassing a wide array of critical healthcare topics like disease conditions, preventive actions, diagnostic tests, treatments, surgeries, and medications. Our extensive evaluation reveals that LMs struggle with factual correctness, demonstrate bias across demographic and linguistic groups, and are susceptible to privacy breaches and adversarial attacks. By highlighting these shortcomings, CLINIC lays the foundation for enhancing the global reach and safety of LMs in healthcare across diverse languages.
694046a1b21c43ffbf4b75bb
Akash Ghosh
claimed_verified
2025-12-16T09:48:43.475Z
630e850c3fc17ffc50f752e5
/avatars/43e4aee228323211b8cf55b783cc54d2.svg
Akash Ghosh
Agcs12
user
Agcs12
694046a1b21c43ffbf4b75bc
Srivarshinee Sridhar
694046a1b21c43ffbf4b75bd
Raghav Kaushik Ravi
694046a1b21c43ffbf4b75be
Muhsin Muhsin
694046a1b21c43ffbf4b75bf
Sriparna Saha
694046a1b21c43ffbf4b75c0
Chirag Agarwal
CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcare
Integrating language models (LMs) in healthcare systems holds great promise for improving medical workflows and decision-making. However, a critical barrier to their real-world adoption is the lack of reliable evaluation of their trustworthiness, especially in multilingual healthcare settings. Existing LMs are predominantly trained in high-resource languages, making them ill-equipped to handle the complexity and diversity of healthcare queries in mid- and low-resource languages, posing significant challenges for deploying them in global healthcare contexts where linguistic diversity is key. In this work, we present CLINIC, a Comprehensive Multilingual Benchmark to evaluate the trustworthiness of language models in healthcare. CLINIC systematically benchmarks LMs across five key dimensions of trustworthiness: truthfulness, fairness, safety, robustness, and privacy, operationalized through 18 diverse tasks, spanning 15 languages (covering all the major continents), and encompassing a wide array of critical healthcare topics like disease conditions, preventive actions, diagnostic tests, treatments, surgeries, and medications. Our extensive evaluation reveals that LMs struggle with factual correctness, demonstrate bias across demographic and linguistic groups, and are susceptible to privacy breaches and adversarial attacks. By highlighting these shortcomings, CLINIC lays the foundation for enhancing the global reach and safety of LMs in healthcare across diverse languages.
CLINIC benchmark evaluates language model trustworthiness in healthcare across multiple dimensions and languages, revealing significant limitations in factual accuracy, bias, and security.
language models multilingual healthcare trustworthiness factual correctness bias privacy breaches adversarial attacks