LLM2Vec-Gen: Generative Embeddings from Large Language Models
LLM-based text embedders typically encode the semantic content of their input. However, embedding tasks require mapping diverse inputs to similar outputs. Typically, this input-output is addressed by training embedding models with paired data using contrastive learning. In this work, we propose a novel self-supervised approach, LLM2Vec-Gen, which adopts a different paradigm: rather than encoding the input, we learn to represent the model's potential response. Specifically, we add trainable special tokens to the LLM's vocabulary, append them to input, and optimize them to represent the LLM's response in a fixed-length sequence. Training is guided by the LLM's own completion for the query, along with an unsupervised embedding teacher that provides distillation targets. This formulation helps to bridge the input-output gap and transfers LLM capabilities such as safety alignment and reasoning to embedding tasks. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 9.3% over the best unsupervised embedding teacher. We also observe up to 43.2% reduction in harmful content retrieval and 29.3% improvement in reasoning capabilities for embedding tasks. Finally, the learned embeddings are interpretable and can be decoded into text to reveal their semantic content.
69b2301c206bad330cd7ff91
Parishad BehnamGhader
claimed_verified
2026-03-12T09:07:03.977Z
627d5ead401f42c57b6ce54c
https://cdn-avatars.huggingface.co/v1/production/uploads/627d5ead401f42c57b6ce54c/GajmN5G_MRUFRZs6ens0t.jpeg
Parishad BehnamGhader
parishadbehnam
user
parishadbehnam
69b2301c206bad330cd7ff92
Vaibhav Adlakha
admin_assigned
2026-03-12T10:09:53.952Z
61981af5d420757268e195ac
/avatars/8b59aaf33447224f83d497425fd7ea8f.svg
Vaibhav Adlakha
vaibhavad
user
vaibhavad
69b2301c206bad330cd7ff93
Fabian David Schmidt
admin_assigned
2026-03-12T10:10:06.945Z
64c8c2d87d0ea4e7f12995c6
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8c2d87d0ea4e7f12995c6/h8eWJrz8kqavemy8vQ2NK.jpeg
Fabian David Schmidt
fdschmidt93
user
fdschmidt93
69b2301c206bad330cd7ff94
Nicolas Chapados
admin_assigned
2026-03-12T10:10:13.029Z
631f54aa5ba8c026340b13cf
https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
Nicolas Chapados
nicolaschapados
user
nicolaschapados
69b2301c206bad330cd7ff95
Marius Mosbach
admin_assigned
2026-03-12T10:10:41.143Z
608865d2511e863acdb20bae
https://cdn-avatars.huggingface.co/v1/production/uploads/608865d2511e863acdb20bae/WSJQHZ4PC9hjLMCWv9mSb.png
Marius Mosbach
mmosbach
user
mmosbach
69b2301c206bad330cd7ff96
Siva Reddy
claimed_verified
2026-03-13T16:50:22.186Z
624734dc4c731bb6bfab8af7
/avatars/6b250b58710a3287b85e4733c1824558.svg
Siva Reddy
sivareddyg
user
sivareddyg
LLM2Vec-Gen: Generative Embeddings from Large Language Models
LLM-based text embedders typically encode the semantic content of their input. However, embedding tasks require mapping diverse inputs to similar outputs. Typically, this input-output is addressed by training embedding models with paired data using contrastive learning. In this work, we propose a novel self-supervised approach, LLM2Vec-Gen, which adopts a different paradigm: rather than encoding the input, we learn to represent the model's potential response. Specifically, we add trainable special tokens to the LLM's vocabulary, append them to input, and optimize them to represent the LLM's response in a fixed-length sequence. Training is guided by the LLM's own completion for the query, along with an unsupervised embedding teacher that provides distillation targets. This formulation helps to bridge the input-output gap and transfers LLM capabilities such as safety alignment and reasoning to embedding tasks. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 9.3% over the best unsupervised embedding teacher. We also observe up to 43.2% reduction in harmful content retrieval and 29.3% improvement in reasoning capabilities for embedding tasks. Finally, the learned embeddings are interpretable and can be decoded into text to reveal their semantic content.
LLM2Vec-Gen introduces a self-supervised method for text embedding that represents model responses through trainable special tokens, achieving superior performance on MTEB while reducing harmful content and improving reasoning capabilities.
LLM-based text embedders contrastive learning self-supervised approach trainable special tokens LLM backbone unsupervised embedding teacher distillation targets Massive Text Embedding Benchmark harmful content retrieval reasoning capabilities