In large-scale machine learning, distributed training commonly involves multiple workers evaluating the gradients of the model on different dataset partitions. A common challenge is the presence of straggling workers, which may significantly slow down training. Traditional gradient coding (GC) addresses this by duplicating dataset partitions across workers, allowing for the replacement of missing gradients from stragglers. However, GC requires workers to evaluate gradients on multiple dataset partitions in each step, potentially increasing overall training time. In this paper, we propose to pipeline GC, such that gradient evaluation is segmented across multiple steps and each worker evaluates gradients on just a single dataset partition per step. We develop the pipelined version for fractional repetition (FR) and cyclic repetition (CR), two representative dataset placement schemes in GC, and prove convergence guarantees for both. Through extensive simulations and experiments on cloud infrastructure, our schemes not only significantly reduce training time but also accelerate convergence compared to GC and other baselines.
Assume \mathbb{E}[g^{(t)}_i] = \nabla F(\beta^{(t)}) for all i. Then:
Assume \mathbb{E}[||g_i^{(s(t,i))}-\nabla F(\beta^{(s(t,i))})||^2_2] \leq \xi\mathbb{E}[||\nabla F(\beta^{(s(t,i))})||^2_2] for some \xi\in[0,1). Then:
Assume that F(\beta) is \lambda-strongly convex and \nabla F(\beta) is L-Lipschitz continuous. Assume the gradient variance is bounded: \mathbb{E}[||g_i^{(s(t,i))}-\nabla F(\beta^{(s(t,i))})||^2_2] \leq \xi\mathbb{E}[||\nabla F(\beta^{(s(t,i))})||^2_2] for some \xi\in[0,1). Assume staleness is bounded: \mathbb{E}[||\nabla F(\beta^{(t)}) - \nabla F(\beta^{(s(t,i))})||^2_2] \leq \gamma \mathbb{E}[||\nabla F(\beta^{(t)})||^2_2] for some \gamma\in[0,1). Then with learning rate \eta \leq \frac{1}{nL(n+\xi)}:
Assume \mathbb{E}[g^{(t)}_i] = \nabla F(\beta^{(t)}) for all i. Then:
Assume \mathbb{E}[||g^{(s(t,i,j))}_i - \nabla F(\beta^{(s(t,i,j))})||^2_2] \leq \xi\mathbb{E}[||\nabla F(\beta^{(s(t,i,j))})||^2_2] for some \xi\in[0,1). Then:
Assume that F(\beta) is \lambda-strongly convex and \nabla F(\beta) is L-Lipschitz continuous. Assume that for some \xi\in[0,1), \mathbb{E}[||g^{(s(t,i,j))}_{i}-\nabla F(\beta^{(s(t,i,j))})||^2_2] \leq \xi \mathbb{E}[||\nabla F(\beta^{(s(t,i,j))})||^2_2]. Define \bar{\nabla}^{(t)} = \frac{1}{nc}\sum_{i=0}^{n-1}\sum_{j=0}^{c-1}\nabla F(\beta^{(s(t,i,j))}). Assume that for some \gamma\in[0,1): \mathbb{E}\!\left[\left|\left|\nabla F(\beta^{(t)})-\bar{\nabla}^{(t)}\right|\right|^2_2\right] \leq \gamma \mathbb{E}[||\nabla F(\beta^{(t)})||^2_2]. Then with learning rate \eta \leq \frac{1}{L(nc+\xi)}:
The text on this page is from arXiv:2607.20739v1 by Xian Su and Jun Li, under the licence CC BY 4.0. The corpus made these changes: it read the items out of the LaTeX source, it expanded the macros of the paper, it put each proof after its statement, and it converted the text to HTML. The formulas are the TeX of the paper, and KaTeX renders them. A formula or a reference can look different from the PDF. The page is published under CC BY 4.0.
The record of each item is in analysis/objects/2607/2607.20739.jsonl.