No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language
  Models
Embedding vision-language models (VLMs) are typically pretrained with short
text windows (<77 tokens), which forces the truncation of long-format captions.
Yet, the distribution of biomedical captions from large-scale open source
literature reveals that a huge portion of captions far exceed 77 tokens. To
this end, we investigate the impact of pretraining on long-format biomedical
captions by extending the context length of text encoders in VLMs. We find that
longer context (thus, enabling additional supervision provided in long-format
captions) correlates with better retrieval and classification performance.
Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M
image-caption pairs enriched with context-aware descriptions from full-text
articles, providing longer and additional textual supervision. Using
BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a
text encoder supporting windows of up to 512 tokens. Our model extends context
capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption
retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in
Recall@1 and +2% average improvements in classification, while also converging
faster than short-context. Our results demonstrate that long-context modeling
is a promising direction for advancing biomedical VLMs.
68e5eb39975ac4c405ef215a
Min Woo Sun
claimed_verified
2025-10-09T06:51:42.299Z
65ac61120844d9e0d67a9f89
https://cdn-avatars.huggingface.co/v1/production/uploads/65ac61120844d9e0d67a9f89/7fK1cfAOtovmUxd3GTvpZ.jpeg
Min Woo Sun
minwoosun
user
minwoosun
68e5eb39975ac4c405ef215b
Alejandro Lozano
68e5eb39975ac4c405ef215c
Javier Gamazo Tejero
68e5eb39975ac4c405ef215d
Vishwesh Nath
68e5eb39975ac4c405ef215e
Xiao Xiao Sun
68e5eb39975ac4c405ef215f
James Burgess
68e5eb39975ac4c405ef2160
Yuhui Zhang
claimed_verified
2026-02-04T12:35:38.032Z
62da55164398e21bf7f0e292
https://cdn-avatars.huggingface.co/v1/production/uploads/62da55164398e21bf7f0e292/xjKkG8IA2IZZqCdjApSh3.jpeg
Yuhui Zhang
yuhuizhang
user
yuhuizhang
68e5eb39975ac4c405ef2161
Kun Yuan
68e5eb39975ac4c405ef2162
Robert Tibshirani
68e5eb39975ac4c405ef2163
Sean Huver
68e5eb39975ac4c405ef2164
Serena Yeung-Levy
No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language
  Models
Embedding vision-language models (VLMs) are typically pretrained with short
text windows (<77 tokens), which forces the truncation of long-format captions.
Yet, the distribution of biomedical captions from large-scale open source
literature reveals that a huge portion of captions far exceed 77 tokens. To
this end, we investigate the impact of pretraining on long-format biomedical
captions by extending the context length of text encoders in VLMs. We find that
longer context (thus, enabling additional supervision provided in long-format
captions) correlates with better retrieval and classification performance.
Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M
image-caption pairs enriched with context-aware descriptions from full-text
articles, providing longer and additional textual supervision. Using
BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a
text encoder supporting windows of up to 512 tokens. Our model extends context
capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption
retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in
Recall@1 and +2% average improvements in classification, while also converging
faster than short-context. Our results demonstrate that long-context modeling
is a promising direction for advancing biomedical VLMs.
Extending the context length of text encoders in vision-language models improves performance on biomedical caption tasks by utilizing longer and more detailed descriptions.
embedding vision-language models VLMs text encoders biomedical captions context length token windows long-format captions BIOMEDICA-LongCAP BMC-LongCLIP long-context modeling Recall@1 classification performance