Unicorn: Text-Only Data Synthesis for Vision Language Model Training
Training vision-language models (VLMs) typically requires large-scale,
high-quality image-text pairs, but collecting or synthesizing such data is
costly. In contrast, text data is abundant and inexpensive, prompting the
question: can high-quality multimodal training data be synthesized purely from
text? To tackle this, we propose a cross-integrated three-stage multimodal data
synthesis framework, which generates two datasets: Unicorn-1.2M and
Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we
construct 1.2M semantically diverse high-quality captions by expanding sparse
caption seeds using large language models (LLMs). In Stage 2:
Instruction-Tuning Data Generation, we further process 471K captions into
multi-turn instruction-tuning tasks to support complex reasoning. Finally, in
Stage 3: Modality Representation Transfer, these textual captions
representations are transformed into visual representations, resulting in
diverse synthetic image representations. This three-stage process enables us to
construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for
instruction-tuning, without relying on real images. By eliminating the
dependency on real images while maintaining data quality and diversity, our
framework offers a cost-effective and scalable solution for VLMs training. Code
is available at https://github.com/Yu-xm/Unicorn.git.
67ebe24d386032554c9e84fc
Xiaomin Yu
claimed_verified
2025-04-02T09:49:42.683Z
64084fa192033c150738e4f2
/avatars/dfff2216eb235c635e5abe6fda3084f0.svg
Yu_xm
Yu2020
user
Yu2020
67ebe24d386032554c9e84fd
Pengxiang Ding
67ebe24d386032554c9e84fe
Wenjie Zhang
67ebe24d386032554c9e84ff
Siteng Huang
claimed_verified
2025-12-11T10:52:46.189Z
65fd82762bf2cd20ddaa193f
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/yBYbWp_mT7UusYdkqtAvw.png
Siteng Huang
huangsiteng
user
huangsiteng
67ebe24d386032554c9e8500
Songyang Gao
67ebe24d386032554c9e8501
Chengwei Qin
67ebe24d386032554c9e8502
Kejian Wu
67ebe24d386032554c9e8503
Zhaoxin Fan
67ebe24d386032554c9e8504
Ziyue Qiao
67ebe24d386032554c9e8505
Donglin Wang
Unicorn: Text-Only Data Synthesis for Vision Language Model Training
Training vision-language models (VLMs) typically requires large-scale,
high-quality image-text pairs, but collecting or synthesizing such data is
costly. In contrast, text data is abundant and inexpensive, prompting the
question: can high-quality multimodal training data be synthesized purely from
text? To tackle this, we propose a cross-integrated three-stage multimodal data
synthesis framework, which generates two datasets: Unicorn-1.2M and
Unicorn-471K-Instruction. In Stage 1: Diverse Caption Data Synthesis, we
construct 1.2M semantically diverse high-quality captions by expanding sparse
caption seeds using large language models (LLMs). In Stage 2:
Instruction-Tuning Data Generation, we further process 471K captions into
multi-turn instruction-tuning tasks to support complex reasoning. Finally, in
Stage 3: Modality Representation Transfer, these textual captions
representations are transformed into visual representations, resulting in
diverse synthetic image representations. This three-stage process enables us to
construct Unicorn-1.2M for pretraining and Unicorn-471K-Instruction for
instruction-tuning, without relying on real images. By eliminating the
dependency on real images while maintaining data quality and diversity, our
framework offers a cost-effective and scalable solution for VLMs training. Code
is available at https://github.com/Yu-xm/Unicorn.git.
A three-stage framework synthesizes high-quality multimodal datasets using text to generate captions and transform them into visual representations, enabling cost-effective training of vision-language models.
vision-language models multimodal data synthesis large language models diverse caption data synthesis instruction-tuning data generation modality representation transfer Unicorn-1.2M Unicorn-471K-Instruction