Method Overview
Integrating denoising diffusion into DETR-based object detection transformers.
(ResNet/Swin)
Encoder
Decoder
Output
DiffuDETR Framework
Overview of the DiffuDETR architecture. The model extracts multi-scale features via a backbone and transformer encoder, then uses noisy reference points (generated from Gaussian noise) along with learnable content queries to produce detections through an iterative diffusion-denoising process in the decoder.
Forward Diffusion
During training, ground-truth bounding box coordinates are corrupted with Gaussian noise at a random timestep $t \sim U(0, 100)$, creating noisy reference points $r_t$ via a cosine noise schedule with only 100 timesteps.
Reverse Denoising
At inference, reference points start as pure Gaussian noise and are iteratively denoised using DDIM sampling. The decoder predicts noise residuals conditioned on image features, requiring only 3 forward passes.
Timestep-Conditioned Decoder
The decoder integrates timestep embeddings after self-attention, conditioning each layer on the current diffusion step: $q_n = \text{FFN}(\text{MSDA}(\text{SA}(q_{n-1}) + t), r_t, O_{\text{enc}})$
Efficient Inference
Only the lightweight decoder is run multiple times—backbone and encoder execute once. With 3 decoder evaluations, the overhead is just ~17% extra FLOPs while achieving optimal performance.
Decoder Architecture
Detailed decoder layer design showing how timestep embeddings are injected after self-attention, followed by multi-scale deformable cross-attention with noisy reference points attending to encoded image features.
$$r_{t-1} = \frac{\sqrt{\bar{\alpha}_{t-1}}}{\sqrt{\bar{\alpha}_t}} \cdot r_t + \left(\sqrt{1 - \bar{\alpha}_{t-1}} - \frac{\sqrt{\bar{\alpha}_{t-1}} \cdot \sqrt{1 - \bar{\alpha}_t}}{\sqrt{\bar{\alpha}_t}}\right) \cdot \hat{\epsilon}_\theta(r_t, t)$$
