Teaching an Agent to Sketch One Part at a Time
We develop a method for producing vector sketches one part at a time. To do this, we train a multi-modal language model-based agent using a novel multi-turn process-reward reinforcement learning following supervised fine-tuning. Our approach is enabled by a new dataset we call ControlSketch-Part, containing rich part-level annotations for sketches, obtained using a novel, generic automatic annotation pipeline that segments vector sketches into semantic parts and assigns paths to parts with a structured multi-stage labeling process. Our results indicate that incorporating structured part-level data and providing agent with the visual feedback through the process enables interpretable, controllable, and locally editable text-to-vector sketch generation.
69c0a2263fd3fef6237a6813
Xiaodan Du
claimed_verified
2026-03-24T09:23:01.154Z
6667839d0209de0f572ea642
https://cdn-avatars.huggingface.co/v1/production/uploads/6667839d0209de0f572ea642/Cjc069UAdv_yCngcyIWU3.jpeg
Xiaodan Du
duxiaodan
user
duxiaodan
69c0a2263fd3fef6237a6814
Ruize Xu
69c0a2263fd3fef6237a6815
David Yunis
69c0a2263fd3fef6237a6816
Yael Vinker
69c0a2263fd3fef6237a6817
Greg Shakhnarovich
Teaching an Agent to Sketch One Part at a Time
We develop a method for producing vector sketches one part at a time. To do this, we train a multi-modal language model-based agent using a novel multi-turn process-reward reinforcement learning following supervised fine-tuning. Our approach is enabled by a new dataset we call ControlSketch-Part, containing rich part-level annotations for sketches, obtained using a novel, generic automatic annotation pipeline that segments vector sketches into semantic parts and assigns paths to parts with a structured multi-stage labeling process. Our results indicate that incorporating structured part-level data and providing agent with the visual feedback through the process enables interpretable, controllable, and locally editable text-to-vector sketch generation.
A multi-modal language model-based agent generates vector sketches incrementally using part-level annotations and process-reward reinforcement learning with visual feedback.
multi-modal language model reinforcement learning supervised fine-tuning ControlSketch-Part vector sketches semantic parts automatic annotation pipeline multi-turn process-reward