LMEB: Long-horizon Memory Embedding Benchmark
Memory embeddings are crucial for memory-augmented systems, such as OpenClaw, but their evaluation is underexplored in current text embedding benchmarks, which narrowly focus on traditional passage retrieval and fail to assess models' ability to handle long-horizon memory retrieval tasks involving fragmented, context-dependent, and temporally distant information. To address this, we introduce the Long-horizon Memory Embedding Benchmark (LMEB), a comprehensive framework that evaluates embedding models' capabilities in handling complex, long-horizon memory retrieval tasks. LMEB spans 22 datasets and 193 zero-shot retrieval tasks across 4 memory types: episodic, dialogue, semantic, and procedural, with both AI-generated and human-annotated data. These memory types differ in terms of level of abstraction and temporal dependency, capturing distinct aspects of memory retrieval that reflect the diverse challenges of the real world. We evaluate 15 widely used embedding models, ranging from hundreds of millions to ten billion parameters. The results reveal that (1) LMEB provides a reasonable level of difficulty; (2) Larger models do not always perform better; (3) LMEB and MTEB exhibit orthogonality. This suggests that the field has yet to converge on a universal model capable of excelling across all memory retrieval tasks, and that performance in traditional passage retrieval may not generalize to long-horizon memory retrieval. In summary, by providing a standardized and reproducible evaluation framework, LMEB fills a crucial gap in memory embedding evaluation, driving further advancements in text embedding for handling long-term, context-dependent memory retrieval. LMEB is available at https://github.com/KaLM-Embedding/LMEB.
69b7593591a28162b233fba7
Xinping Zhao
claimed_verified
2026-03-16T10:40:25.371Z
658c0b0574e79b9a8e9de89a
https://cdn-avatars.huggingface.co/v1/production/uploads/658c0b0574e79b9a8e9de89a/8HeMEOT5cLEzauXGlrqF_.jpeg
Xinping Zhao
Yuki131
user
Yuki131
69b7593591a28162b233fba8
Xinshuo Hu
admin_assigned
2026-03-16T10:51:43.298Z
62de46a83af42b53b1cb6931
https://cdn-avatars.huggingface.co/v1/production/uploads/62de46a83af42b53b1cb6931/jV7xzQFcqy-8-xouXy2FN.jpeg
Xinshuo Hu
YanshekWoo
user
YanshekWoo
69b7593591a28162b233fba9
Jiaxin Xu
69b7593591a28162b233fbaa
Danyu Tang
69b7593591a28162b233fbab
Xin Zhang
69b7593591a28162b233fbac
Mengjia Zhou
69b7593591a28162b233fbad
Yan Zhong
69b7593591a28162b233fbae
Yao Zhou
69b7593591a28162b233fbaf
Zifei Shan
admin_assigned
2026-03-16T10:53:18.326Z
66b1bbc8642b9e6d7a6e6db2
https://cdn-avatars.huggingface.co/v1/production/uploads/66b1bbc8642b9e6d7a6e6db2/SajUVqX7jNg3PM1qKYyU2.png
Zifei Shan
zifeishan
user
zifeishan
69b7593591a28162b233fbb0
Meishan Zhang
69b7593591a28162b233fbb1
Baotian Hu
69b7593591a28162b233fbb2
Min Zhang
LMEB: Long-horizon Memory Embedding Benchmark
Memory embeddings are crucial for memory-augmented systems, such as OpenClaw, but their evaluation is underexplored in current text embedding benchmarks, which narrowly focus on traditional passage retrieval and fail to assess models' ability to handle long-horizon memory retrieval tasks involving fragmented, context-dependent, and temporally distant information. To address this, we introduce the Long-horizon Memory Embedding Benchmark (LMEB), a comprehensive framework that evaluates embedding models' capabilities in handling complex, long-horizon memory retrieval tasks. LMEB spans 22 datasets and 193 zero-shot retrieval tasks across 4 memory types: episodic, dialogue, semantic, and procedural, with both AI-generated and human-annotated data. These memory types differ in terms of level of abstraction and temporal dependency, capturing distinct aspects of memory retrieval that reflect the diverse challenges of the real world. We evaluate 15 widely used embedding models, ranging from hundreds of millions to ten billion parameters. The results reveal that (1) LMEB provides a reasonable level of difficulty; (2) Larger models do not always perform better; (3) LMEB and MTEB exhibit orthogonality. This suggests that the field has yet to converge on a universal model capable of excelling across all memory retrieval tasks, and that performance in traditional passage retrieval may not generalize to long-horizon memory retrieval. In summary, by providing a standardized and reproducible evaluation framework, LMEB fills a crucial gap in memory embedding evaluation, driving further advancements in text embedding for handling long-term, context-dependent memory retrieval. LMEB is available at https://github.com/KaLM-Embedding/LMEB.
A new benchmark evaluates embedding models' ability to handle long-horizon memory retrieval tasks, revealing that performance in traditional passage retrieval does not generalize to complex memory retrieval scenarios.
memory embeddings memory-augmented systems text embedding benchmarks long-horizon memory retrieval embedding models passage retrieval memory types episodic memory dialogue memory semantic memory procedural memory zero-shot retrieval MTEB parameter-efficient fine-tuning