Introducing VLRM: Vision-Language Models act as Reward Models for Image Captioning
text
Introducing VLRM: Vision-Language Models act as Reward Models for Image Captioning
new_line
Paper:
text
Paper:
new_line
https://arxiv.org/abs/2404.01911
https://arxiv.org/abs/2404.01911
link
new_line
Model weights and training logs:
text
Model weights and training logs:
new_line
https://huggingface.co/sashakunitsyn/vlrm-blip2-opt-2.7b
sashakunitsyn/vlrm-blip2-opt-2.7b
model
resource
https://huggingface.co/sashakunitsyn/vlrm-blip2-opt-2.7b
new_line
new_line
We propose VLRM, a method for fine-tuning an existing image captioning model using reinforcement learning and vision-language models as reward models.
text
We propose VLRM, a method for fine-tuning an existing image captioning model using reinforcement learning and vision-language models as reward models.
new_line
The method manages to significantly improve the generation quality without human-labeled data and is applicable to any image captioning model.
text
The method manages to significantly improve the generation quality without human-labeled data and is applicable to any image captioning model.
new_line
Our model reaches impressive 0.90 R@1 CLIP Recall score on MS-COCO Carpathy Test Split.
text
Our model reaches impressive 0.90 R@1 CLIP Recall score on MS-COCO Carpathy Test Split.
Introducing VLRM: Vision-Language Models act as Reward Models for Image Captioning
Paper:
https://arxiv.org/abs/2404.01911
Model weights and training logs:
https://huggingface.co/sashakunitsyn/vlrm-blip2-opt-2.7b

We propose VLRM, a method for fine-tuning an existing image captioning model using reinforcement learning and vision-language models as reward models.
The method manages to significantly improve the generation quality without human-labeled data and is applicable to any image captioning model.
Our model reaches impressive 0.90 R@1 CLIP Recall score on MS-COCO Carpathy Test Split.
❤️
samusenps nikgr victor clem osanseviero merve Kukedlc AlkaouDembele xziayro
➕
samusenps merve