Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained
  Classification
Text-to-image (T2I) models are increasingly used for synthetic dataset
generation, but generating effective synthetic training data for classification
remains challenging. Fine-tuning a T2I model with a few real examples can help
improve the quality of synthetic training data; however, it may also cause
overfitting and reduce diversity in the generated samples. We propose a
fine-tuning strategy BOB (BeyondOBjects) to mitigate these concerns for
fine-grained classification. Given a small set of real examples, we first
extract class-agnostic attributes such as scene background and object pose. We
then explicitly condition on these attributes during fine-tuning of the T2I
model and marginalize them out during generation. This design mitigates
overfitting, preserves the T2I model's generative prior, reduces estimation
errors, and further minimizes unintended inter-class associations. Extensive
experiments across multiple T2I models, backbones, and datasets show that our
method achieves state-of-the-art performance in low-shot fine-grained
classification when augmented with synthetic data. Concretely, BOB outperforms
DataDream by 7.4% on the Aircraft dataset (from 50.0% to 57.4% when fine-tuning
a CLIP classifier with five real images augmented with 100 synthetic images).
In three of the four benchmarks, fine-tuning downstream models with 5 real
images augmented with BOB achieves better performance than fine-tuning with 10
real images. Collectively, BOB outperforms prior art in 18 of 24 experimental
settings, with 2+% accuracy improvements in 14 of these settings.
690919a6812eca10f9cc6110
William Yang
claimed_verified
2025-11-04T20:26:57.434Z
66c7b4aeb5ed2bcc2706f978
/avatars/ffa5dcab545715c037f4a2fa7c5ffbab.svg
william yang
yang-william
user
yang-william
690919a6812eca10f9cc6111
Xindi Wu
claimed_verified
2026-02-20T08:38:38.318Z
613940c0905b1938233881e3
https://cdn-avatars.huggingface.co/v1/production/uploads/613940c0905b1938233881e3/Vb4kFWKEq6AILUP9KmCZA.png
Xindi Wu
xindiw
user
xindiw
690919a6812eca10f9cc6112
Zhiwei Deng
690919a6812eca10f9cc6113
Esin Tureci
690919a6812eca10f9cc6114
Olga Russakovsky
Beyond Objects: Contextual Synthetic Data Generation for Fine-Grained
  Classification
Text-to-image (T2I) models are increasingly used for synthetic dataset
generation, but generating effective synthetic training data for classification
remains challenging. Fine-tuning a T2I model with a few real examples can help
improve the quality of synthetic training data; however, it may also cause
overfitting and reduce diversity in the generated samples. We propose a
fine-tuning strategy BOB (BeyondOBjects) to mitigate these concerns for
fine-grained classification. Given a small set of real examples, we first
extract class-agnostic attributes such as scene background and object pose. We
then explicitly condition on these attributes during fine-tuning of the T2I
model and marginalize them out during generation. This design mitigates
overfitting, preserves the T2I model's generative prior, reduces estimation
errors, and further minimizes unintended inter-class associations. Extensive
experiments across multiple T2I models, backbones, and datasets show that our
method achieves state-of-the-art performance in low-shot fine-grained
classification when augmented with synthetic data. Concretely, BOB outperforms
DataDream by 7.4% on the Aircraft dataset (from 50.0% to 57.4% when fine-tuning
a CLIP classifier with five real images augmented with 100 synthetic images).
In three of the four benchmarks, fine-tuning downstream models with 5 real
images augmented with BOB achieves better performance than fine-tuning with 10
real images. Collectively, BOB outperforms prior art in 18 of 24 experimental
settings, with 2+% accuracy improvements in 14 of these settings.
A fine-tuning strategy called BOB improves synthetic data quality for low-shot fine-grained classification by conditioning on class-agnostic attributes and marginalizing them during generation.
text-to-image models fine-tuning class-agnostic attributes scene background object pose generative prior estimation errors unintended inter-class associations low-shot fine-grained classification DataDream CLIP classifier synthetic data