Locally Initialized LoRA Gradient Descent
Descriptions may be stale. Consult the linked Lean sources.
Locally Initialized LoRA Gradient Descent
We consider minimizing a differentiable, strongly convex function
Write
Our assumptions:
Assumption 1 (Strong convexity and smoothness). The
function
Assumption 2 (Minimizing factors and local
initialization). There exist factors
Our result is algorithm-specific: we analyze LoRA gradient descent
with simultaneous updates of both factors. Starting from the given
Results
Let
| Result | Oracle complexity | Novelty | Limitations |
|---|---|---|---|
| (1) SilverLoraLocalConvergence (Ours) (review) | |||
| (2) Constant-step LoRA-GD (LongTails, n.d.). Derived bound (in-repo). | (1): Accuracy coefficient
|
(1): Worse at moderate
accuracy: |
|
| (3) Ambient AGD (Bubeck 2015,
Thm. 3.18) plus rank- |
Same gradient oracle and target; outside scalar-step factor GD. |
(1): Method-specific local silver-step guarantee within scalar LoRA-GD; no established complexity improvement. | (1): Never a smaller
bound; polynomially worse as |
| (4) Factored AGD (Li and Lin 2019, Thm. 4, §7). Local initialization and index-set constraints of Thm. 4. | Restricted strong convexity; balancing regularizer, momentum and restarts. |
(1): Rates not comparable
in |
(1): Requires global rather than restricted strong convexity. |
Abbr: Costs count full-gradient calls.