Adam under Generalized Smoothness
Descriptions may be stale. Consult the linked Lean sources.
Adam under Generalized Smoothness
We consider minimizing a differentiable, possibly nonconvex function
The optimization problem is
Our assumptions:
Assumption 1 (Generalized smoothness). The function
Assumption 2 (Initial objective gap). For every
Assumption 3 (Stochastic gradient oracle). Let
Our result is algorithm-specific: we analyze Adam with the following
recurrence. The algorithm chooses constants
The required output is measurable, its squared gradient norm is
integrable, and it satisfies
Results
Differentiable
| Result | Oracle complexity | Novelty | Limitations |
|---|---|---|---|
| (1) ExpectedGeneralizedSmoothAdam
(Ours) |
|||
| (2) Adam (Jin et al. 2026),
Thm. B.2; |
(1): Scope extends the
same Adam expectation guarantee to |
(1) needs IID bounded
variance instead of conditional, gap/gradient-dependent second moments;
fixes damping |
|
| (3) Adam (Jin et al. 2026), Claim B.1; globally smooth special case | (1): Smaller explicit
polynomial parameter powers on the globally smooth overlap; matches the
|
(1) requires IID noise; (3) permits conditional bounded variance. | |
| (4) Adam (Jin et al. 2026),
Thm. B.1; |
(1): Expected RMS for one
fixed run; (4) tunes steps to |
(1) needs IID bounded variance instead of conditional, gap/gradient-dependent second moments. | |
| (5) Adam (Zhang et al. 2025), Thm. 2; global coordinatewise smoothness | (1): Local Euclidean
instead of global coordinatewise smoothness (allows |
(1) requires IID bounded variance; (5) allows conditional coordinatewise affine variance and uses damping inside the square root. | |
| (6) NSGD-M (Hübler et al. 2024),
Cor. 16; |
(1): RMS certificate. For
the shared mean target, noise dependence is |
(1) has dimension
dependence and worse smoothness/gap powers ( |
Abbr: Costs count individual stochastic-gradient
calls. RMS means