Optimal Stopping in Latent Diffusion Model

In brief

• Problem. The last steps of latent diffusion can degrade samples, which does not happen in pixel-space diffusion.
• Approach. A Gaussian model with linear autoencoders that links latent dimension, the constraints of score matching and the stopping time.
• Result. Low-dimensional latents call for earlier stopping, high-dimensional ones for later; stopping time is a key hyperparameter of latent diffusion.

Abstract

We identify and analyze a surprising phenomenon of Latent Diffusion Models (LDMs) where the final steps of the diffusion can degrade sample quality. In contrast to conventional arguments that justify early stopping for numerical stability, this phenomenon is intrinsic to the dimensionality reduction in LDMs. We provide a principled explanation by analyzing the interaction between latent dimension and stopping time. Under a Gaussian framework with linear autoencoders, we characterize the conditions under which early stopping is needed to minimize the distance between generated and target distributions. More precisely, we show that lower-dimensional representations benefit from earlier termination, whereas higher-dimensional latent spaces require later stopping time. We further establish that the latent dimension interplays with other hyperparameters of the problem such as constraints in the parameters of score matching. Experiments on synthetic and real datasets illustrate these properties, underlining that early stopping can improve generative quality. Together, our results offer a theoretical foundation for understanding how the latent dimension influences the sample quality, and highlight stopping time as a key hyperparameter in LDMs.

Cite

Yu-Han Wu, Quentin Berthet, Gérard Biau, Claire Boyer, Romuald Elie and Pierre Marion (2026). Optimal Stopping in Latent Diffusion Models. In International Conference on Machine Learning (ICML).

@inproceedings{wu2026optimal,
  title={Optimal Stopping in Latent Diffusion Models},
  author={Wu, Yu-Han and Berthet, Quentin and Biau, G{\'e}rard and Boyer, Claire and Elie, Romuald and Marion, Pierre},
  booktitle={International Conference on Machine Learning ({ICML})},
  year={2026}
}

Related work

• MIND: Monge Inception Distance for Generative Models Evaluation NeurIPS 2026 A sliced-Wasserstein alternative to FID: more sample-efficient, faster, and more robust.
• Understanding diffusion models requires rethinking (again) generalization NeurIPS 2026 position track Generalization in diffusion models needs new theory: what is learned before memorization?