D2D: Detector-to-Differentiable Critic for Improved Numeracy in
  Text-to-Image Generation
Text-to-image (T2I) diffusion models have achieved strong performance in
semantic alignment, yet they still struggle with generating the correct number
of objects specified in prompts. Existing approaches typically incorporate
auxiliary counting networks as external critics to enhance numeracy. However,
since these critics must provide gradient guidance during generation, they are
restricted to regression-based models that are inherently differentiable, thus
excluding detector-based models with superior counting ability, whose
count-via-enumeration nature is non-differentiable. To overcome this
limitation, we propose Detector-to-Differentiable (D2D), a novel framework that
transforms non-differentiable detection models into differentiable critics,
thereby leveraging their superior counting ability to guide numeracy
generation. Specifically, we design custom activation functions to convert
detector logits into soft binary indicators, which are then used to optimize
the noise prior at inference time with pre-trained T2I models. Our extensive
experiments on SDXL-Turbo, SD-Turbo, and Pixart-DMD across four benchmarks of
varying complexity (low-density, high-density, and multi-object scenarios)
demonstrate consistent and substantial improvements in object counting accuracy
(e.g., boosting up to 13.7% on D2D-Small, a 400-prompt, low-density benchmark),
with minimal degradation in overall image quality and computational overhead.
69095f1c812eca10f9cc6163
Nobline Yoo
claimed_verified
2025-11-04T20:26:53.098Z
69095e06b8c21a27dbc0d2b6
/avatars/3c5f5c1140024c084bf8c8d935dd3d90.svg
Nobline Yoo
n-yoo
user
n-yoo
69095f1c812eca10f9cc6164
Olga Russakovsky
69095f1c812eca10f9cc6165
Ye Zhu
D2D: Detector-to-Differentiable Critic for Improved Numeracy in
  Text-to-Image Generation
Text-to-image (T2I) diffusion models have achieved strong performance in
semantic alignment, yet they still struggle with generating the correct number
of objects specified in prompts. Existing approaches typically incorporate
auxiliary counting networks as external critics to enhance numeracy. However,
since these critics must provide gradient guidance during generation, they are
restricted to regression-based models that are inherently differentiable, thus
excluding detector-based models with superior counting ability, whose
count-via-enumeration nature is non-differentiable. To overcome this
limitation, we propose Detector-to-Differentiable (D2D), a novel framework that
transforms non-differentiable detection models into differentiable critics,
thereby leveraging their superior counting ability to guide numeracy
generation. Specifically, we design custom activation functions to convert
detector logits into soft binary indicators, which are then used to optimize
the noise prior at inference time with pre-trained T2I models. Our extensive
experiments on SDXL-Turbo, SD-Turbo, and Pixart-DMD across four benchmarks of
varying complexity (low-density, high-density, and multi-object scenarios)
demonstrate consistent and substantial improvements in object counting accuracy
(e.g., boosting up to 13.7% on D2D-Small, a 400-prompt, low-density benchmark),
with minimal degradation in overall image quality and computational overhead.
A novel framework, Detector-to-Differentiable (D2D), transforms non-differentiable detection models into differentiable critics to improve object counting accuracy in text-to-image diffusion models with minimal impact on image quality.
text-to-image (T2I) diffusion models semantic alignment auxiliary counting networks gradient guidance detector-based models non-differentiable Detector-to-Differentiable (D2D) custom activation functions soft binary indicators noise prior SDXL-Turbo SD-Turbo Pixart-DMD object counting accuracy image quality computational overhead