SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes
Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces. Current scene synthesis methods produce sparsely furnished rooms that lack the dense clutter, articulated furniture, and physical properties essential for robotic manipulation. We introduce SceneSmith, a hierarchical agentic framework that generates simulation-ready indoor environments from natural language prompts. SceneSmith constructs scenes through successive stagesx2013from architectural layout to furniture placement to small object populationx2013each implemented as an interaction among VLM agents: designer, critic, and orchestrator. The framework tightly integrates asset generation through text-to-3D synthesis for static objects, dataset retrieval for articulated objects, and physical property estimation. SceneSmith generates 3-6x more objects than prior methods, with <2% inter-object collisions and 96% of objects remaining stable under physics simulation. In a user study with 205 participants, it achieves 92% average realism and 91% average prompt faithfulness win rates against baselines. We further demonstrate that these environments can be used in an end-to-end pipeline for automatic robot policy evaluation.
698bfc136052d3bed9630ae7
Nicholas Pfaff
claimed_verified
2026-02-11T11:17:17.749Z
663d126ee6257fa86ac793eb
/avatars/954f47804a9149dfc1c1d5de4dfefb63.svg
Nicholas Pfaff
nepfaff
user
nepfaff
698bfc136052d3bed9630ae8
Thomas Cohn
698bfc136052d3bed9630ae9
Sergey Zakharov
698bfc136052d3bed9630aea
Rick Cory
698bfc136052d3bed9630aeb
Russ Tedrake
SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes
Simulation has become a key tool for training and evaluating home robots at scale, yet existing environments fail to capture the diversity and physical complexity of real indoor spaces. Current scene synthesis methods produce sparsely furnished rooms that lack the dense clutter, articulated furniture, and physical properties essential for robotic manipulation. We introduce SceneSmith, a hierarchical agentic framework that generates simulation-ready indoor environments from natural language prompts. SceneSmith constructs scenes through successive stagesx2013from architectural layout to furniture placement to small object populationx2013each implemented as an interaction among VLM agents: designer, critic, and orchestrator. The framework tightly integrates asset generation through text-to-3D synthesis for static objects, dataset retrieval for articulated objects, and physical property estimation. SceneSmith generates 3-6x more objects than prior methods, with <2% inter-object collisions and 96% of objects remaining stable under physics simulation. In a user study with 205 participants, it achieves 92% average realism and 91% average prompt faithfulness win rates against baselines. We further demonstrate that these environments can be used in an end-to-end pipeline for automatic robot policy evaluation.
SceneSmith is a hierarchical agentic framework that generates simulation-ready indoor environments from natural language prompts through multiple stages involving VLM agents and integrated asset generation techniques.
SceneSmith hierarchical agentic framework natural language prompts VLM agents text-to-3D synthesis dataset retrieval physical property estimation simulation-ready environments robotic manipulation end-to-end pipeline