Outlier tokens in Diffusion Transformers distort attention mechanisms and degrade image generation quality. Researchers analyze how high-norm tokens emerge in both the encoder and denoiser stages of modern pipelines, revealing patterns that hinder model stability. This study provides developers with crucial insights for stabilizing DiT training and optimizing transformer-based generative architectures.
Opening Kapyn…