Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory
Large language model (LLM) agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence. Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.
69a8fcba8b5b27c7ea9fa64e
Zhenting Wang
admin_assigned
2026-03-05T10:28:01.539Z
64dfcc62e8b6f3f3baa950e0
/avatars/21bbff67d46c08044efe2406575aa77e.svg
Zhenting Wang
ztwang
user
ztwang
69a8fcba8b5b27c7ea9fa64f
Huancheng Chen
69a8fcba8b5b27c7ea9fa650
Jiayun Wang
claimed_verified
2026-03-09T10:38:10.377Z
64f0fe244fcec1a8fde793d6
/avatars/30347f81ba6d9b85007af7c29fb14b47.svg
peter wang
samaonline
user
samaonline
69a8fcba8b5b27c7ea9fa651
Wei Wei
Memex(RL): Scaling Long-Horizon LLM Agents via Indexed Experience Memory
Large language model (LLM) agents are fundamentally bottlenecked by finite context windows on long-horizon tasks. As trajectories grow, retaining tool outputs and intermediate reasoning in-context quickly becomes infeasible: the working context becomes prohibitively long, eventually exceeds the context budget, and makes distant evidence harder to use even when it is still present. Existing solutions typically shorten context through truncation or running summaries, but these methods are fundamentally lossy because they compress or discard past evidence itself. We introduce Memex, an indexed experience memory mechanism that instead compresses context without discarding evidence. Memex maintains a compact working context consisting of concise structured summaries and stable indices, while storing full-fidelity underlying interactions in an external experience database under those indices. The agent can then decide when to dereference an index and recover the exact past evidence needed for the current subgoal. We optimize both write and read behaviors with our reinforcement learning framework MemexRL, using reward shaping tailored to indexed memory usage under a context budget, so the agent learns what to summarize, what to archive, how to index it, and when to retrieve it. This yields a substantially less lossy form of long-horizon memory than summary-only approaches. We further provide a theoretical analysis showing the potential of the Memex loop to preserve decision quality with bounded dereferencing while keeping effective in-context computation bounded as history grows. Empirically, on challenging long-horizon tasks, Memex agent trained with MemexRL improves task success while using a significantly smaller working context.
A memory mechanism called Memex enables large language model agents to handle long-horizon tasks more effectively by maintaining compact context through structured summaries while storing full interaction details in an external database, allowing selective retrieval based on learned criteria.
large language model agents context windows experience memory indexed memory reinforcement learning reward shaping memory compression context budget dereferencing decision quality in-context computation