RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction
As Large Language Models (LLMs) evolve from static dialogue interfaces to autonomous general agents, effective memory is paramount to ensuring long-term consistency. However, existing benchmarks primarily focus on casual conversation or task-oriented dialogue, failing to capture **"long-term project-oriented"** interactions where agents must track evolving goals.
  To bridge this gap, we introduce **RealMem**, the first benchmark grounded in realistic project scenarios. RealMem comprises over 2,000 cross-session dialogues across eleven scenarios, utilizing natural user queries for evaluation.
  We propose a synthesis pipeline that integrates Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to simulate the dynamic evolution of memory. Experiments reveal that current memory systems face significant challenges in managing the long-term project states and dynamic context dependencies inherent in real-world projects.
  Our code and datasets are available at [https://github.com/AvatarMemory/RealMemBench](https://github.com/AvatarMemory/RealMemBench).
69663f3987c71000b5a910ab
Haonan Bian
claimed_verified
2026-01-13T15:44:54.957Z
661e2c508289f9d60d6088a3
/avatars/0a588a091ca20a1da5acabf9feb8d8b1.svg
bhn
Haonan-Bian
user
Haonan-Bian
69663f3987c71000b5a910ac
Zhiyuan Yao
69663f3987c71000b5a910ad
Sen Hu
69663f3987c71000b5a910ae
Zishan Xu
69663f3987c71000b5a910af
Shaolei Zhang
69663f3987c71000b5a910b0
Yifu Guo
69663f3987c71000b5a910b1
Ziliang Yang
69663f3987c71000b5a910b2
Xueran Han
69663f3987c71000b5a910b3
Huacan Wang
claimed_verified
2026-01-15T15:06:06.864Z
6603d56ab4344a2b07cd6d21
/avatars/1569bb60166532317c85e80da722ba1c.svg
Huacan Wang
Huacan-Wang
user
Huacan-Wang
69663f3987c71000b5a910b4
Ronghao Chen
RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction
As Large Language Models (LLMs) evolve from static dialogue interfaces to autonomous general agents, effective memory is paramount to ensuring long-term consistency. However, existing benchmarks primarily focus on casual conversation or task-oriented dialogue, failing to capture **"long-term project-oriented"** interactions where agents must track evolving goals.
  To bridge this gap, we introduce **RealMem**, the first benchmark grounded in realistic project scenarios. RealMem comprises over 2,000 cross-session dialogues across eleven scenarios, utilizing natural user queries for evaluation.
  We propose a synthesis pipeline that integrates Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to simulate the dynamic evolution of memory. Experiments reveal that current memory systems face significant challenges in managing the long-term project states and dynamic context dependencies inherent in real-world projects.
  Our code and datasets are available at [https://github.com/AvatarMemory/RealMemBench](https://github.com/AvatarMemory/RealMemBench).
RealMem benchmark evaluates memory systems for long-term project-oriented interactions in large language models, revealing challenges in managing dynamic context dependencies.
Large Language Models memory systems project-oriented interactions cross-session dialogues multi-agent dialogue generation memory management dynamic context dependencies