CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare
Multimodal agentic pipelines are transforming human-computer interaction by enabling efficient and accessible automation of complex, real-world tasks. However, recent efforts have focused on short-horizon or general-purpose applications (e.g., mobile or desktop interfaces), leaving long-horizon automation for domain-specific systems, particularly in healthcare, largely unexplored. To address this, we introduce CareFlow, a high-quality human-annotated benchmark comprising complex, long-horizon software workflows across medical annotation tools, DICOM viewers, EHR systems, and laboratory information systems. On this benchmark, existing vision-language models (VLMs) perform poorly, struggling with long-horizon reasoning and multi-step interactions in medical contexts. To overcome this, we propose CarePilot, a multi-agent framework based on the actor-critic paradigm. The Actor integrates tool grounding with dual-memory mechanisms (long-term and short-term experience) to predict the next semantic action from the visual interface and system state. The Critic evaluates each action, updates memory based on observed effects, and either executes or provides corrective feedback to refine the workflow. Through iterative agentic simulation, the Actor learns to perform more robust and reasoning-aware predictions during inference. Our experiments show that CarePilot achieves state-of-the-art performance, outperforming strong closed-source and open-source multimodal baselines by approximately 15.26% and 3.38%, respectively, on our benchmark and out-of-distribution dataset.
69c48fc54ae7eef815b9d92e
Akash Ghosh
claimed_verified
2026-03-26T13:43:07.448Z
630e850c3fc17ffc50f752e5
/avatars/43e4aee228323211b8cf55b783cc54d2.svg
Akash Ghosh
Agcs12
user
Agcs12
69c48fc54ae7eef815b9d92f
Tajamul Ashraf
69c48fc54ae7eef815b9d930
Rishu Kumar Singh
69c48fc54ae7eef815b9d931
Numan Saeed
69c48fc54ae7eef815b9d932
Sriparna Saha
69c48fc54ae7eef815b9d933
Xiuying Chen
69c48fc54ae7eef815b9d934
Salman Khan
CarePilot: A Multi-Agent Framework for Long-Horizon Computer Task Automation in Healthcare
Multimodal agentic pipelines are transforming human-computer interaction by enabling efficient and accessible automation of complex, real-world tasks. However, recent efforts have focused on short-horizon or general-purpose applications (e.g., mobile or desktop interfaces), leaving long-horizon automation for domain-specific systems, particularly in healthcare, largely unexplored. To address this, we introduce CareFlow, a high-quality human-annotated benchmark comprising complex, long-horizon software workflows across medical annotation tools, DICOM viewers, EHR systems, and laboratory information systems. On this benchmark, existing vision-language models (VLMs) perform poorly, struggling with long-horizon reasoning and multi-step interactions in medical contexts. To overcome this, we propose CarePilot, a multi-agent framework based on the actor-critic paradigm. The Actor integrates tool grounding with dual-memory mechanisms (long-term and short-term experience) to predict the next semantic action from the visual interface and system state. The Critic evaluates each action, updates memory based on observed effects, and either executes or provides corrective feedback to refine the workflow. Through iterative agentic simulation, the Actor learns to perform more robust and reasoning-aware predictions during inference. Our experiments show that CarePilot achieves state-of-the-art performance, outperforming strong closed-source and open-source multimodal baselines by approximately 15.26% and 3.38%, respectively, on our benchmark and out-of-distribution dataset.
careflow is a benchmark for long-horizon automation in healthcare, and carepilot is a multimodal agent framework that uses actor-critic methods with dual-memory mechanisms to improve automated task execution in complex medical environments.
vision-language models multimodal agentic pipelines actor-critic paradigm tool grounding dual-memory mechanisms long-horizon reasoning multi-step interactions automated task execution