UM-Text: A Unified Multimodal Model for Image Understanding
With the rapid advancement of image generation, visual text editing using natural language instructions has received increasing attention. The main challenge of this task is to fully understand the instruction and reference image, and thus generate visual text that is style-consistent with the image. Previous methods often involve complex steps of specifying the text content and attributes, such as font size, color, and layout, without considering the stylistic consistency with the reference image. To address this, we propose UM-Text, a unified multimodal model for context understanding and visual text editing by natural language instructions. Specifically, we introduce a Visual Language Model (VLM) to process the instruction and reference image, so that the text content and layout can be elaborately designed according to the context information. To generate an accurate and harmonious visual text image, we further propose the UM-Encoder to combine the embeddings of various condition information, where the combination is automatically configured by VLM according to the input instruction. During training, we propose a regional consistency loss to offer more effective supervision for glyph generation on both latent and RGB space, and design a tailored three-stage training strategy to further enhance model performance. In addition, we contribute the UM-DATA-200K, a large-scale visual text image dataset on diverse scenes for model training. Extensive qualitative and quantitative results on multiple public benchmarks demonstrate that our method achieves state-of-the-art performance.
6967daf0c5e371f6b235d328
Lichen Ma
6967daf0c5e371f6b235d329
Xiaolong Fu
claimed_verified
2026-01-15T15:03:51.089Z
670ce8badeb97b8b72e6e6e6
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/_x-e95IYnkVQLb2KV7nIv.png
fuxiaolong
fxlong
user
fxlong
6967daf0c5e371f6b235d32a
Gaojing Zhou
6967daf0c5e371f6b235d32b
Zipeng Guo
6967daf0c5e371f6b235d32c
Ting Zhu
6967daf0c5e371f6b235d32d
Yichun Liu
6967daf0c5e371f6b235d32e
Yu Shi
6967daf0c5e371f6b235d32f
Jason Li
6967daf0c5e371f6b235d330
Junshi Huang
UM-Text: A Unified Multimodal Model for Image Understanding
With the rapid advancement of image generation, visual text editing using natural language instructions has received increasing attention. The main challenge of this task is to fully understand the instruction and reference image, and thus generate visual text that is style-consistent with the image. Previous methods often involve complex steps of specifying the text content and attributes, such as font size, color, and layout, without considering the stylistic consistency with the reference image. To address this, we propose UM-Text, a unified multimodal model for context understanding and visual text editing by natural language instructions. Specifically, we introduce a Visual Language Model (VLM) to process the instruction and reference image, so that the text content and layout can be elaborately designed according to the context information. To generate an accurate and harmonious visual text image, we further propose the UM-Encoder to combine the embeddings of various condition information, where the combination is automatically configured by VLM according to the input instruction. During training, we propose a regional consistency loss to offer more effective supervision for glyph generation on both latent and RGB space, and design a tailored three-stage training strategy to further enhance model performance. In addition, we contribute the UM-DATA-200K, a large-scale visual text image dataset on diverse scenes for model training. Extensive qualitative and quantitative results on multiple public benchmarks demonstrate that our method achieves state-of-the-art performance.
A unified multimodal model for visual text editing that understands natural language instructions and maintains stylistic consistency with reference images through visual language modeling and contextual embedding combination.
visual language model visual text editing natural language instructions stylistic consistency UM-Encoder regional consistency loss three-stage training strategy UM-DATA-200K