Add or edit metadata

Title

Unicode, TeX accent characters, and MathJax are supported. See title formatting help.

Authors

Use GivenName(s) FamilyName(s) or I. FamilyName. Separate individual authors with a comma or the word “and”. See author name formatting help.

fake author

Abstract

Limit of 1920 characters. Unicode, TeX accent characters, and MathJax are supported. See abstract formatting help.

Abstract: We study the convergence behaviour of stochastic gradient methods on non-convex objectives arising in deep network training. Existing analyses typically assume bounded gradient noise, an assumption that is violated in practice by the heavy-tailed gradient distributions observed in transformer architectures. We relax this assumption and prove convergence under a weaker moment condition, obtaining rates that degrade gracefully as the tail index decreases.<br>Our analysis introduces a coupling argument that relates the discrete iterates to a continuous-time surrogate, avoiding the uniform boundedness requirements of prior work. We further show that gradient clipping, widely used as a \em heuristic, admits a principled interpretation in this framework as a bias-variance trade-off controlled by the clipping threshold. Data and code are available at https://example.org/heavy-tail-sgd. Empirically, we validate the predicted rates on language modelling and image classification benchmarks, and find that the measured tail indices fall within the regime our theory covers.

Comments(optional)

Supplemental information such as number of pages or figures, conference information.

DOI(optional)

Full DOI of the version of record.

Journal reference(optional)

See journal reference help for details.

Report number(optional)

See report number help for details.

ACM classification(optional)

Example: F.2.2; I.2.7

MSC classification(optional)

Example: 14J60 (Primary), 14F05, 14J26 (Secondary)