Experience Transfer for Multimodal LLM Agents in Minecraft Game
Multimodal LLM agents operating in complex game environments must continually reuse past experience to solve new tasks efficiently. In this work, we propose Echo, a transfer-oriented memory framework that enables agents to derive actionable knowledge from prior interactions rather than treating memory as a passive repository of static records. To make transfer explicit, Echo decomposes reusable knowledge into five dimensions: structure, attribute, process, function, and interaction. This formulation allows the agent to identify recurring patterns shared across different tasks and infer what prior experience remains applicable in new situations. Building on this formulation, Echo leverages In-Context Analogy Learning (ICAL) to retrieve relevant experiences and adapt them to unseen tasks through contextual examples. Experiments in Minecraft show that, under a from-scratch learning setting, Echo achieves a 1.3x to 1.7x speed-up on object-unlocking tasks. Moreover, Echo exhibits a burst-like chain-unlocking phenomenon, rapidly unlocking multiple similar items within a short time interval after acquiring transferable experience. These results suggest that experience transfer is a promising direction for improving the efficiency and adaptability of multimodal LLM agents in complex interactive environments.
69d5c3fcb9b236b710ae8186
Chenghao Li
69d5c3fcb9b236b710ae8187
Jun Liu
69d5c3fcb9b236b710ae8188
Songbo Zhang
69d5c3fcb9b236b710ae8189
Huadong Jian
69d5c3fcb9b236b710ae818a
Hao Ni
69d5c3fcb9b236b710ae818b
Lik-Hang Lee
69d5c3fcb9b236b710ae818c
Sung-Ho Bae
69d5c3fcb9b236b710ae818d
Guoqing Wang
69d5c3fcb9b236b710ae818e
Yang Yang
69d5c3fcb9b236b710ae818f
Chaoning Zhang
Experience Transfer for Multimodal LLM Agents in Minecraft Game
Multimodal LLM agents operating in complex game environments must continually reuse past experience to solve new tasks efficiently. In this work, we propose Echo, a transfer-oriented memory framework that enables agents to derive actionable knowledge from prior interactions rather than treating memory as a passive repository of static records. To make transfer explicit, Echo decomposes reusable knowledge into five dimensions: structure, attribute, process, function, and interaction. This formulation allows the agent to identify recurring patterns shared across different tasks and infer what prior experience remains applicable in new situations. Building on this formulation, Echo leverages In-Context Analogy Learning (ICAL) to retrieve relevant experiences and adapt them to unseen tasks through contextual examples. Experiments in Minecraft show that, under a from-scratch learning setting, Echo achieves a 1.3x to 1.7x speed-up on object-unlocking tasks. Moreover, Echo exhibits a burst-like chain-unlocking phenomenon, rapidly unlocking multiple similar items within a short time interval after acquiring transferable experience. These results suggest that experience transfer is a promising direction for improving the efficiency and adaptability of multimodal LLM agents in complex interactive environments.
Echo is a transfer-oriented memory framework for multimodal LLM agents that enables efficient task solving in complex game environments by deriving actionable knowledge from prior interactions through decomposition into five knowledge dimensions and In-Context Analogy Learning.
multimodal LLM agents transfer-oriented memory framework In-Context Analogy Learning knowledge decomposition experience transfer recurrent patterns contextual examples Minecraft object-unlocking tasks chain-unlocking phenomenon