🎉 PaliGemma fine-tuned in Flickr30K Translated for Brazilian Portuguese Image Captioning

PaliGemma (google/paligemma-3b-pt-224) model fine-tuned for image captioning on Flickr30K Portuguese (translated version using Google Translator API).

🤖 Model Description

🧑‍💻 How to Get Started with the Model

Use the code below to get started with the model.

Install libraries:

pip install transformers==4.45.2 bitsandbytes==0.45.2 peft==0.13.2

Python code:

import requests
import torch
from PIL import Image

from transformers import PaliGemmaProcessor, PaliGemmaForConditionalGeneration, BitsAndBytesConfig
from huggingface_hub import login

# Use your HuggingFace API key, since Paligemma is available through user form submission
login('hf_...')

# load a fine-tuned image captioning model, and corresponding tokenizer and image processor
model = PaliGemmaForConditionalGeneration.from_pretrained(
    'google/paligemma-3b-pt-224',
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type='nf4',
        bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    device_map={'':0}
)
model.load_adapter('laicsiifes/paligemma-flickr30k_pt')
processor = PaliGemmaProcessor.from_pretrained('laicsiifes/paligemma-flickr30k_pt', trust_remote_code=True)

# preprocess an image
image = Image.open(requests.get("http://images.cocodataset.org/val2014/COCO_val2014_000000458153.jpg", stream=True).raw)
inputs = processor(
    text='caption pt\n',
    images=image,
    return_tensors='pt'
).to('cuda:0')

# generate caption
generated_ids = model.generate(**inputs, max_new_tokens=25)
prediction = generated_ids[:, inputs['input_ids'].shape[1]:].tolist()
generated_text = processor.batch_decode(prediction, skip_special_tokens=True)[0]

import matplotlib.pyplot as plt

# plot image with caption
plt.imshow(image)
plt.axis("off")
plt.title(generated_text)
plt.show()

📈 Results

The evaluation metrics: CIDEr-D, BLEU@4, ROUGE-L, METEOR, BERTScore (using BERTimbau), and CLIP-Score (using CAPIVARA).

Model	#Params	CIDEr	BLEU-4	ROUGE-L	METEOR	BERTScore	CLIP-Score
ViTucano 1B	1.53B	78.78	26.29	46.67	50.25	73.44	55.48
ViTucano 2B	2.88B	80.03	27.32	47.31	50.90	73.69	56.16
PaliGemma	2.92B	47.25	18.63	39.69	48.68	69.87	60.13
Phi-3 V	4.15B	75.47	27.26	47.24	48.24	73.20	55.44
LLaMa 3.2 V	11.70B	70.94	23.97	45.05	47.13	72.71	55.65