The Pensieve Paradigm: Stateful Language Models Mastering Their Own Context
In the world of Harry Potter, when Dumbledore's mind is overburdened, he extracts memories into a Pensieve to be revisited later. In the world of AI, while we possess the Pensieve-mature databases and retrieval systems, our models inexplicably lack the "wand" to operate it. They remain like a Dumbledore without agency, passively accepting a manually engineered context as their entire memory. This work finally places the wand in the model's hand. We introduce StateLM, a new class of foundation models endowed with an internal reasoning loop to manage their own state. We equip our model with a suite of memory tools, such as context pruning, document indexing, and note-taking, and train it to actively manage these tools. By learning to dynamically engineering its own context, our model breaks free from the architectural prison of a fixed window. Experiments across various model sizes demonstrate StateLM's effectiveness across diverse scenarios. On long-document QA tasks, StateLMs consistently outperform standard LLMs across all model scales; on the chat memory task, they achieve absolute accuracy improvements of 10% to 20% over standard LLMs. On the deep research task BrowseComp-Plus, the performance gap becomes even more pronounced: StateLM achieves up to 52% accuracy, whereas standard LLM counterparts struggle around 5%. Ultimately, our approach shifts LLMs from passive predictors to state-aware agents where reasoning becomes a stateful and manageable process.
698e8affcace060ff123abfa
Xiaoyuan Liu
698e8affcace060ff123abfb
Tian Liang
698e8affcace060ff123abfc
Dongyang Ma
698e8affcace060ff123abfd
Deyu Zhou
698e8affcace060ff123abfe
Haitao Mi
698e8affcace060ff123abff
Pinjia He
698e8affcace060ff123ac00
Yan Wang
claimed_verified
2026-02-13T09:37:03.071Z
64674375946476c5d215382d
/avatars/0f0b98080c064a17d003da8a22a047cc.svg
Yan Wang
libertywing
user
libertywing
The Pensieve Paradigm: Stateful Language Models Mastering Their Own Context
In the world of Harry Potter, when Dumbledore's mind is overburdened, he extracts memories into a Pensieve to be revisited later. In the world of AI, while we possess the Pensieve-mature databases and retrieval systems, our models inexplicably lack the "wand" to operate it. They remain like a Dumbledore without agency, passively accepting a manually engineered context as their entire memory. This work finally places the wand in the model's hand. We introduce StateLM, a new class of foundation models endowed with an internal reasoning loop to manage their own state. We equip our model with a suite of memory tools, such as context pruning, document indexing, and note-taking, and train it to actively manage these tools. By learning to dynamically engineering its own context, our model breaks free from the architectural prison of a fixed window. Experiments across various model sizes demonstrate StateLM's effectiveness across diverse scenarios. On long-document QA tasks, StateLMs consistently outperform standard LLMs across all model scales; on the chat memory task, they achieve absolute accuracy improvements of 10% to 20% over standard LLMs. On the deep research task BrowseComp-Plus, the performance gap becomes even more pronounced: StateLM achieves up to 52% accuracy, whereas standard LLM counterparts struggle around 5%. Ultimately, our approach shifts LLMs from passive predictors to state-aware agents where reasoning becomes a stateful and manageable process.
StateLM enables language models to actively manage their own memory and context through internal reasoning loops and memory tools, significantly improving performance on long-document tasks and chat memory challenges.
foundation models internal reasoning loop memory tools context pruning document indexing note-taking dynamic context engineering long-document QA chat memory task deep research task BrowseComp-Plus