CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates
Large-scale Vision-Language Models (VLMs) exhibit impressive complex reasoning capabilities but remain largely unexplored in visual sequential planning, i.e., executing multi-step actions towards a goal. Additionally, practical sequential planning often involves non-optimal (erroneous) steps, challenging VLMs to detect and correct such steps. We propose Corrective Sequential Planning Benchmark (CoSPlan) to evaluate VLMs in error-prone, vision-based sequential planning tasks across 4 domains: maze navigation, block rearrangement, image reconstruction,and object reorganization. CoSPlan assesses two key abilities: Error Detection (identifying non-optimal action) and Step Completion (correcting and completing action sequences to reach the goal). Despite using state-of-the-art reasoning techniques such as Chain-of-Thought and Scene Graphs, VLMs (e.g. Intern-VLM and Qwen2) struggle on CoSPlan, failing to leverage contextual cues to reach goals. Addressing this, we propose a novel training-free method, Scene Graph Incremental updates (SGI), which introduces intermediate reasoning steps between the initial and goal states. SGI helps VLMs reason about sequences, yielding an average performance gain of 5.2%. In addition to enhancing reliability in corrective sequential planning, SGI generalizes to traditional planning tasks such as Plan-Bench and VQA.
693df666f516c69324680db8
Shresth Grover
693df666f516c69324680db9
Priyank Pathak
claimed_verified
2025-12-15T08:12:58.446Z
682ce2c990497b25a9851b6e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/_i7hSIyQhL9eFdM2TBAev.webp
Priyank Pathak
ppriyank
user
ppriyank
693df666f516c69324680dba
Akash Kumar
693df666f516c69324680dbb
Vibhav Vineet
693df666f516c69324680dbc
Yogesh S Rawat
CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates
Large-scale Vision-Language Models (VLMs) exhibit impressive complex reasoning capabilities but remain largely unexplored in visual sequential planning, i.e., executing multi-step actions towards a goal. Additionally, practical sequential planning often involves non-optimal (erroneous) steps, challenging VLMs to detect and correct such steps. We propose Corrective Sequential Planning Benchmark (CoSPlan) to evaluate VLMs in error-prone, vision-based sequential planning tasks across 4 domains: maze navigation, block rearrangement, image reconstruction,and object reorganization. CoSPlan assesses two key abilities: Error Detection (identifying non-optimal action) and Step Completion (correcting and completing action sequences to reach the goal). Despite using state-of-the-art reasoning techniques such as Chain-of-Thought and Scene Graphs, VLMs (e.g. Intern-VLM and Qwen2) struggle on CoSPlan, failing to leverage contextual cues to reach goals. Addressing this, we propose a novel training-free method, Scene Graph Incremental updates (SGI), which introduces intermediate reasoning steps between the initial and goal states. SGI helps VLMs reason about sequences, yielding an average performance gain of 5.2%. In addition to enhancing reliability in corrective sequential planning, SGI generalizes to traditional planning tasks such as Plan-Bench and VQA.
VLMs struggle with error-prone vision-based sequential planning tasks, but Scene Graph Incremental updates (SGI) improves their performance by introducing intermediate reasoning steps.
Vision-Language Models VLMs Chain-of-Thought Scene Graphs Corrective Sequential Planning Benchmark CoSPlan Error Detection Step Completion Scene Graph Incremental updates SGI Plan-Bench VQA