Canvas-to-Image: Compositional Image Generation with Multimodal Controls
While modern diffusion models excel at generating high-quality and diverse images, they still struggle with high-fidelity compositional and multimodal control, particularly when users simultaneously specify text prompts, subject references, spatial arrangements, pose constraints, and layout annotations. We introduce Canvas-to-Image, a unified framework that consolidates these heterogeneous controls into a single canvas interface, enabling users to generate images that faithfully reflect their intent. Our key idea is to encode diverse control signals into a single composite canvas image that the model can directly interpret for integrated visual-spatial reasoning. We further curate a suite of multi-task datasets and propose a Multi-Task Canvas Training strategy that optimizes the diffusion model to jointly understand and integrate heterogeneous controls into text-to-image generation within a unified learning paradigm. This joint training enables Canvas-to-Image to reason across multiple control modalities rather than relying on task-specific heuristics, and it generalizes well to multi-control scenarios during inference. Extensive experiments show that Canvas-to-Image significantly outperforms state-of-the-art methods in identity preservation and control adherence across challenging benchmarks, including multi-person composition, pose-controlled composition, layout-constrained generation, and multi-control generation.
6927cfb1243b2216fb75cd97
Yusuf Dalva
claimed_verified
2025-11-30T20:10:27.019Z
65454d7c117ecae648892170
/avatars/83a7091a24bf86801176ca85234b417a.svg
Yusuf Dalva
ydalva
user
ydalva
6927cfb1243b2216fb75cd98
Guocheng Gordon Qian
6927cfb1243b2216fb75cd99
Maya Goldenberg
6927cfb1243b2216fb75cd9a
Tsai-Shien Chen
claimed_verified
2025-12-15T08:14:19.892Z
6330f570b68c7453d2ee8945
https://cdn-avatars.huggingface.co/v1/production/uploads/6330f570b68c7453d2ee8945/PwmHigwGHsSAl3v6QCWe8.jpeg
Tsai-Shien Chen
tsaishien-chen
user
tsaishien-chen
6927cfb1243b2216fb75cd9b
Kfir Aberman
6927cfb1243b2216fb75cd9c
Sergey Tulyakov
6927cfb1243b2216fb75cd9d
Pinar Yanardag
6927cfb1243b2216fb75cd9e
Kuan-Chieh Jackson Wang
Canvas-to-Image: Compositional Image Generation with Multimodal Controls
While modern diffusion models excel at generating high-quality and diverse images, they still struggle with high-fidelity compositional and multimodal control, particularly when users simultaneously specify text prompts, subject references, spatial arrangements, pose constraints, and layout annotations. We introduce Canvas-to-Image, a unified framework that consolidates these heterogeneous controls into a single canvas interface, enabling users to generate images that faithfully reflect their intent. Our key idea is to encode diverse control signals into a single composite canvas image that the model can directly interpret for integrated visual-spatial reasoning. We further curate a suite of multi-task datasets and propose a Multi-Task Canvas Training strategy that optimizes the diffusion model to jointly understand and integrate heterogeneous controls into text-to-image generation within a unified learning paradigm. This joint training enables Canvas-to-Image to reason across multiple control modalities rather than relying on task-specific heuristics, and it generalizes well to multi-control scenarios during inference. Extensive experiments show that Canvas-to-Image significantly outperforms state-of-the-art methods in identity preservation and control adherence across challenging benchmarks, including multi-person composition, pose-controlled composition, layout-constrained generation, and multi-control generation.
Canvas-to-Image is a unified framework that encodes diverse control signals into a composite canvas image for high-fidelity multimodal image generation, outperforming existing methods in various benchmarks.
diffusion models high-fidelity compositional multimodal control text prompts subject references spatial arrangements pose constraints layout annotations unified framework composite canvas image visual-spatial reasoning multi-task datasets Multi-Task Canvas Training text-to-image generation identity preservation control adherence multi-person composition pose-controlled composition layout-constrained generation multi-control generation