VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
Modern Vision-Language Models (VLMs) remain poorly characterized in multi-step visual interactions, particularly in how they integrate perception, memory, and action over long horizons. We introduce VisGym, a gymnasium of 17 environments for evaluating and training VLMs. The suite spans symbolic puzzles, real-image understanding, navigation, and manipulation, and provides flexible controls over difficulty, input representation, planning horizon, and feedback. We also provide multi-step solvers that generate structured demonstrations, enabling supervised finetuning. Our evaluations show that all frontier models struggle in interactive settings, achieving low success rates in both the easy (46.6%) and hard (26.0%) configurations. Our experiments reveal notable limitations: models struggle to effectively leverage long context, performing worse with an unbounded history than with truncated windows. Furthermore, we find that several text-based symbolic tasks become substantially harder once rendered visually. However, explicit goal observations, textual feedback, and exploratory demonstrations in partially observable or unknown-dynamics settings for supervised finetuning yield consistent gains, highlighting concrete failure modes and pathways for improving multi-step visual decision-making. Code, data, and models can be found at: https://visgym.github.io/.
6976d4695d41524304c13523
Zirui Wang
claimed_verified
2026-01-26T08:28:54.291Z
641a38fdfb5ffff5ac78ceb0
https://cdn-avatars.huggingface.co/v1/production/uploads/641a38fdfb5ffff5ac78ceb0/53nGstZ9Ya5WfeGdXAXCr.png
Zirui Wang
zwcolin
user
zwcolin
6976d4695d41524304c13524
Junyi Zhang
claimed_verified
2026-01-27T09:43:49.929Z
62f0ecd2700bdc19558360de
/avatars/5325b4b763f30c41f30e3aec0d2b59fa.svg
Junyi Zhang
Junyi42
user
Junyi42
6976d4695d41524304c13525
Jiaxin Ge
claimed_verified
2026-01-26T08:31:33.595Z
6629dac35e13d8145e3a605e
/avatars/95938f20ab9e067838f37aca6ea235ae.svg
Jiaxin Ge
JiaxinGe
user
JiaxinGe
6976d4695d41524304c13526
Long Lian
claimed_verified
2026-01-26T08:28:52.034Z
63797c273f575acc2f6893c0
/avatars/32d7a6a8881c8c4d80a097b732ed24b6.svg
Long(Tony) Lian
longlian
user
longlian
6976d4695d41524304c13527
Letian Fu
6976d4695d41524304c13528
Lisa Dunlap
6976d4695d41524304c13529
Ken Goldberg
6976d4695d41524304c1352a
XuDong Wang
6976d4695d41524304c1352b
Ion Stoica
6976d4695d41524304c1352c
David M. Chan
6976d4695d41524304c1352d
Sewon Min
6976d4695d41524304c1352e
Joseph E. Gonzalez
VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
Modern Vision-Language Models (VLMs) remain poorly characterized in multi-step visual interactions, particularly in how they integrate perception, memory, and action over long horizons. We introduce VisGym, a gymnasium of 17 environments for evaluating and training VLMs. The suite spans symbolic puzzles, real-image understanding, navigation, and manipulation, and provides flexible controls over difficulty, input representation, planning horizon, and feedback. We also provide multi-step solvers that generate structured demonstrations, enabling supervised finetuning. Our evaluations show that all frontier models struggle in interactive settings, achieving low success rates in both the easy (46.6%) and hard (26.0%) configurations. Our experiments reveal notable limitations: models struggle to effectively leverage long context, performing worse with an unbounded history than with truncated windows. Furthermore, we find that several text-based symbolic tasks become substantially harder once rendered visually. However, explicit goal observations, textual feedback, and exploratory demonstrations in partially observable or unknown-dynamics settings for supervised finetuning yield consistent gains, highlighting concrete failure modes and pathways for improving multi-step visual decision-making. Code, data, and models can be found at: https://visgym.github.io/.
Modern vision-language models exhibit significant challenges in multi-step visual interaction tasks, particularly in long-horizon perception-memory-action integration, with performance declining when handling unbounded historical contexts.
vision-language models multi-step visual interactions perception memory action symbolic puzzles real-image understanding navigation manipulation supervised fine-tuning goal observations textual feedback exploratory demonstrations partially observable environments unknown-dynamics settings