Silver Lora Local Convergence — Locally Initialized LoRA Gradient Descent
Descriptions may be stale. Consult the linked Lean sources.
Locally Initialized LoRA Gradient Descent · Lean proof
Locally Initialized LoRA
Gradient Descent
We consider minimizing a differentiable, strongly convex function
with Lipschitz gradient through LoRA factors
, where are positive integers and . A global minimizer
admits full-rank factors
with controlled conditioning, and the given starting factors lie in a
prescribed neighborhood of those factors. The oracle returns at . For , the goal is final
factors
satisfying
almost surely under , the law of
the algorithm’s randomness.
Write
and . We study Matrix
spaces carry the Frobenius inner product and norm. The factor space
carries the product norm Vector
norms are Euclidean, and denotes the
induced operator norm. The algorithm receives , the parameters , and the
starting factors ,
with
Our assumptions:
Assumption 1 (Strong convexity and smoothness). The
function is differentiable
everywhere on . For all ,
Assumption 2 (Minimizing factors and local
initialization). There exist factors
such that
is a global minimizer of on : Define the linear map by
These same factors satisfy
Our result is algorithm-specific: we analyze LoRA gradient descent
with simultaneous updates of both factors. Starting from the given , each iteration queries the product
and uses the returned matrix
in The scalar step size must be strictly positive for
every history and every oracle reply; it may depend on the algorithm’s
randomness, its accumulated information, and the current reply. After
oracle calls, the output is
the final factor pair ,
and it must satisfy
Proved result
OracleTemplate.Convergence LongTails.Setups.LocalSmoothStronglyConvexLora.spec
LongTails.Theorems.SilverLoraLocalConvergence.budget
Relevant previous results
No qualifying prior reference verified; this is not a proof of novelty.