Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution
Task scheduling is critical for embodied AI, enabling agents to follow natural language instructions and execute actions efficiently in 3D physical worlds. However, existing datasets often simplify task planning by ignoring operations research (OR) knowledge and 3D spatial grounding. In this work, we propose Operations Research knowledge-based 3D Grounded Task Scheduling (ORS3D), a new task that requires the synergy of language understanding, 3D grounding, and efficiency optimization. Unlike prior settings, ORS3D demands that agents minimize total completion time by leveraging parallelizable subtasks, e.g., cleaning the sink while the microwave operates. To facilitate research on ORS3D, we construct ORS3D-60K, a large-scale dataset comprising 60K composite tasks across 4K real-world scenes. Furthermore, we propose GRANT, an embodied multi-modal large language model equipped with a simple yet effective scheduling token mechanism to generate efficient task schedules and grounded actions. Extensive experiments on ORS3D-60K validate the effectiveness of GRANT across language understanding, 3D grounding, and scheduling efficiency. The code is available at https://github.com/H-EmbodVis/GRANT
69271171243b2216fb75cc51
Dingkang Liang
69271171243b2216fb75cc52
Cheng Zhang
claimed_verified
2025-11-27T09:59:30.501Z
6478657c91398856110aba40
https://cdn-avatars.huggingface.co/v1/production/uploads/6478657c91398856110aba40/SwWRmWozOWiC4aoLSuRqD.jpeg
Cheng Zhang
Younai
user
Younai
69271171243b2216fb75cc53
Xiaopeng Xu
69271171243b2216fb75cc54
Jianzhong Ju
69271171243b2216fb75cc55
Zhenbo Luo
69271171243b2216fb75cc56
Xiang Bai
Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution
Task scheduling is critical for embodied AI, enabling agents to follow natural language instructions and execute actions efficiently in 3D physical worlds. However, existing datasets often simplify task planning by ignoring operations research (OR) knowledge and 3D spatial grounding. In this work, we propose Operations Research knowledge-based 3D Grounded Task Scheduling (ORS3D), a new task that requires the synergy of language understanding, 3D grounding, and efficiency optimization. Unlike prior settings, ORS3D demands that agents minimize total completion time by leveraging parallelizable subtasks, e.g., cleaning the sink while the microwave operates. To facilitate research on ORS3D, we construct ORS3D-60K, a large-scale dataset comprising 60K composite tasks across 4K real-world scenes. Furthermore, we propose GRANT, an embodied multi-modal large language model equipped with a simple yet effective scheduling token mechanism to generate efficient task schedules and grounded actions. Extensive experiments on ORS3D-60K validate the effectiveness of GRANT across language understanding, 3D grounding, and scheduling efficiency. The code is available at https://github.com/H-EmbodVis/GRANT
ORS3D, a new task requiring language understanding, 3D grounding, and efficient scheduling, is introduced with a large dataset and an embodied multi-modal model named GRANT that uses a scheduling token mechanism for effective task management.
Operations Research knowledge-based 3D Grounded Task Scheduling ORS3D 3D grounding efficiency optimization parallelizable subtasks embodied multi-modal large language model GRANT scheduling token mechanism