A Comparative Study in Surgical AI: Datasets, Foundation Models, and Barriers to Med-AGI
Recent Artificial Intelligence (AI) models have matched or exceeded human experts in several benchmarks of biomedical task performance, but have lagged behind on surgical image-analysis benchmarks. Since surgery requires integrating disparate tasks -- including multimodal data integration, human interaction, and physical effects -- generally-capable AI models could be particularly attractive as a collaborative tool if performance could be improved. On the one hand, the canonical approach of scaling architecture size and training data is attractive, especially since there are millions of hours of surgical video data generated per year. On the other hand, preparing surgical data for AI training requires significantly higher levels of professional expertise, and training on that data requires expensive computational resources. These trade-offs paint an uncertain picture of whether and to-what-extent modern AI could aid surgical practice. In this paper, we explore this question through a case study of surgical tool detection using state-of-the-art AI methods available in 2026. We demonstrate that even with multi-billion parameter models and extensive training, current Vision Language Models fall short in the seemingly simple task of tool detection in neurosurgery. Additionally, we show scaling experiments indicating that increasing model size and training time only leads to diminishing improvements in relevant performance metrics. Thus, our experiments suggest that current models could still face significant obstacles in surgical use cases. Moreover, some obstacles cannot be simply ``scaled away'' with additional compute and persist across diverse model architectures, raising the question of whether data and label availability are the only limiting factors. We discuss the main contributors to these constraints and advance potential solutions.
69cbcbb8b45993d253952599
Kirill Skobelev
claimed_verified
2026-04-01T08:35:51.174Z
671150795f07bf2a6035ee95
/avatars/610c940caec2bdb92a92a556648b0c0c.svg
Kirill Skobelev
skblv
user
skblv
69cbcbb8b45993d25395259a
Eric Fithian
69cbcbb8b45993d25395259b
Yegor Baranovski
69cbcbb8b45993d25395259c
Jack Cook
69cbcbb8b45993d25395259d
Sandeep Angara
69cbcbb8b45993d25395259e
Shauna Otto
69cbcbb8b45993d25395259f
Zhuang-Fang Yi
69cbcbb8b45993d2539525a0
John Zhu
69cbcbb8b45993d2539525a1
Daniel A. Donoho
69cbcbb8b45993d2539525a2
X. Y. Han
69cbcbb8b45993d2539525a3
Neeraj Mainkar
69cbcbb8b45993d2539525a4
Margaux Masson-Forsythe
A Comparative Study in Surgical AI: Datasets, Foundation Models, and Barriers to Med-AGI
Recent Artificial Intelligence (AI) models have matched or exceeded human experts in several benchmarks of biomedical task performance, but have lagged behind on surgical image-analysis benchmarks. Since surgery requires integrating disparate tasks -- including multimodal data integration, human interaction, and physical effects -- generally-capable AI models could be particularly attractive as a collaborative tool if performance could be improved. On the one hand, the canonical approach of scaling architecture size and training data is attractive, especially since there are millions of hours of surgical video data generated per year. On the other hand, preparing surgical data for AI training requires significantly higher levels of professional expertise, and training on that data requires expensive computational resources. These trade-offs paint an uncertain picture of whether and to-what-extent modern AI could aid surgical practice. In this paper, we explore this question through a case study of surgical tool detection using state-of-the-art AI methods available in 2026. We demonstrate that even with multi-billion parameter models and extensive training, current Vision Language Models fall short in the seemingly simple task of tool detection in neurosurgery. Additionally, we show scaling experiments indicating that increasing model size and training time only leads to diminishing improvements in relevant performance metrics. Thus, our experiments suggest that current models could still face significant obstacles in surgical use cases. Moreover, some obstacles cannot be simply ``scaled away'' with additional compute and persist across diverse model architectures, raising the question of whether data and label availability are the only limiting factors. We discuss the main contributors to these constraints and advance potential solutions.
Current vision language models struggle with surgical tool detection despite large-scale training, indicating fundamental limitations beyond compute and data availability.
Vision Language Models surgical tool detection neurosurgery multi-billion parameter models surgical image-analysis multimodal data integration human interaction physical effects