Talk ENS Lyon · Lyon, France
COLT 2025
Oral presentation of Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization.
Slides
The paper behind this talk
Taking a Big Step: Large Learning Rates in Denoising Score Matching Prevent Memorization
COLT 2025
- Problem. The exact minimizer of denoising score matching memorizes the training data, yet trained models memorize only mildly.
- Approach. Analyze gradient descent with large steps, which can only converge to minima whose sharpness is bounded by the inverse learning rate.
- Result. The memorizing solution is too sharp to be reached; large learning rates prevent memorization.
