LMK > CLS: Landmark Pooling for Dense Embeddings
Representation learning is central to many downstream tasks such as search, clustering, classification, and reranking. State-of-the-art sequence encoders typically collapse a variable-length token sequence to a single vector using a pooling operator, most commonly a special [CLS] token or mean pooling over token embeddings. In this paper, we identify systematic weaknesses of these pooling strategies: [CLS] tends to concentrate information toward the initial positions of the sequence and can under-represent distributed evidence, while mean pooling can dilute salient local signals, sometimes leading to worse short-context performance. To address these issues, we introduce Landmark (LMK) pooling, which partitions a sequence into chunks, inserts landmark tokens between chunks, and forms the final representation by mean-pooling the landmark token embeddings. This simple mechanism improves long-context extrapolation without sacrificing local salient features, at the cost of introducing a small number of special tokens. We empirically demonstrate that LMK pooling matches existing methods on short-context retrieval tasks and yields substantial improvements on long-context tasks, making it a practical and scalable alternative to existing pooling methods.
697c3614a67238fac88cc19a
Meet Doshi
claimed_verified
2026-01-30T09:35:41.547Z
649e9dbdb24e556c33b9e508
https://cdn-avatars.huggingface.co/v1/production/uploads/649e9dbdb24e556c33b9e508/B4M6OjrzL7Ac-vZFjydm_.jpeg
Meet Doshi
meetdoshi90
user
meetdoshi90
697c3614a67238fac88cc19b
Aashka Trivedi
697c3614a67238fac88cc19c
Vishwajeet Kumar
claimed_verified
2026-02-02T16:59:35.335Z
62c45be1143622c9278f1ae2
/avatars/0ddc94e2e8428334f85f13e6f08a66c2.svg
vishwajeet kumar
vishwajeetkumar
user
vishwajeetkumar
697c3614a67238fac88cc19d
Parul Awasthy
697c3614a67238fac88cc19e
Yulong Li
697c3614a67238fac88cc19f
Jaydeep Sen
697c3614a67238fac88cc1a0
Radu Florian
697c3614a67238fac88cc1a1
Sachindra Joshi
LMK > CLS: Landmark Pooling for Dense Embeddings
Representation learning is central to many downstream tasks such as search, clustering, classification, and reranking. State-of-the-art sequence encoders typically collapse a variable-length token sequence to a single vector using a pooling operator, most commonly a special [CLS] token or mean pooling over token embeddings. In this paper, we identify systematic weaknesses of these pooling strategies: [CLS] tends to concentrate information toward the initial positions of the sequence and can under-represent distributed evidence, while mean pooling can dilute salient local signals, sometimes leading to worse short-context performance. To address these issues, we introduce Landmark (LMK) pooling, which partitions a sequence into chunks, inserts landmark tokens between chunks, and forms the final representation by mean-pooling the landmark token embeddings. This simple mechanism improves long-context extrapolation without sacrificing local salient features, at the cost of introducing a small number of special tokens. We empirically demonstrate that LMK pooling matches existing methods on short-context retrieval tasks and yields substantial improvements on long-context tasks, making it a practical and scalable alternative to existing pooling methods.
Landmark pooling improves long-context representation learning by partitioning sequences into chunks and using landmark tokens to preserve both global and local information more effectively than traditional pooling methods.
representation learning sequence encoders pooling operator [CLS] token mean pooling landmark pooling token sequences long-context extrapolation short-context performance