Design fault-tolerant training for a frontier model on 16k accelerators

Train a large model (say 400B parameters on 15T tokens) on about 16,000 accelerators for about two months. At that scale hardware fails every few hours, and every failure stops the whole synchronous job. Design the system that keeps the run going: how work is split across chips, how state is saved and restored, how failures and bad numerics are detected and handled, and how much compute is lost.

Functional requirements

• Run synchronous data, tensor and pipeline parallel training across thousands of hosts.
• Save training state regularly and resume from it after any failure, landing on exactly the same data position.
• Detect failed or slow hardware, replace it, and restart automatically, at any hour.
• Detect numerical problems (loss spikes, NaNs, silent data corruption) and recover.
• Give researchers visibility: loss curves, throughput, goodput, incidents.

Non-functional requirements

• Goodput (useful training time divided by wall time) above 90%.
• Recovery from a single host failure in minutes, without a person.
• Reproducibility: resuming gives the same result as not stopping, up to floating-point nondeterminism.

Back-of-the-envelope estimation

• Compute: about 6 × N × D = 6 × 4e11 × 1.5e13 = 3.6e25 FLOPs. 16,384 chips at about 1e15 bf16 FLOP/s and 40% model FLOPs utilization deliver 6.5e18 FLOP/s: 5.5M s, about 64 days. Each 1% of goodput is about 15 hours of the whole cluster.
• Failures: if one chip (with its share of host, network and memory) fails about once every 50,000 hours, 16,384 of them fail about every 50,000 / 16,384 ≈ 3 hours. Roughly 8 interruptions a day.
• Training state: about 16 bytes per parameter (bf16 weights, fp32 master weights, two fp32 Adam moments) = 6.4 TB. Sharded over 16,384 chips: about 390 MB per chip.
• Checkpoint stall: copying 390 MB from device to host memory takes well under a second; with coordination it is a few seconds of stalled training. Uploading 6.4 TB to object storage at an aggregate 500 GB/s takes about 13 s, in the background.

Internal interfaces between components:

• controller.launch(job_spec) / controller.restart(job_id, from_step)
• worker.save(step) -> shard_ref (async; returns when the host-memory snapshot is taken)
• worker.restore(step, shard_ref); data_loader.seek(step)
• monitor.report(rank, step, step_time_ms, loss, grad_norm, health) every step or every few steps

• checkpoints(job_id, step, shards[rank -> uri, checksum], data_state, rng_state, complete: bool, created_at); a checkpoint is usable only when every shard is written and complete is set (written last).
• incidents(job_id, step, kind, ranks, hosts, detected_at, recovered_at, lost_seconds)
• hosts(id, state: active | spare | cordoned, error_counts, last_heartbeat)

Build it one requirement at a time. Each drawing adds to the one before; the shaded boxes are what that step adds.

1) Split the model across the cluster

A controller launches one worker process per chip. Work is split three ways: tensor parallel across the 8 chips of a node (fastest links), pipeline parallel across 16 stages, and sharded data parallel across the remaining 128 groups (8 × 16 × 128 = 16,384). A deterministic data loader feeds each data-parallel group its batches.

{"type":"excalidraw","elements":[
 {"type":"rectangle","id":"ctrl","x":0,"y":140,"width":180,"height":70,"label":{"text":"Controller"}},
 {"type":"rectangle","id":"w1","x":340,"y":0,"width":180,"height":70,"label":{"text":"Workers 1"}},
 {"type":"rectangle","id":"w2","x":340,"y":140,"width":180,"height":70,"label":{"text":"Workers 2"}},
 {"type":"rectangle","id":"w3","x":340,"y":280,"width":180,"height":70,"label":{"text":"Workers N"}},
 {"type":"ellipse","id":"data","x":670,"y":-5,"width":220,"height":80,"label":{"text":"Training data"}},
 {"type":"arrow","id":"a1","x":180,"y":160,"width":160,"height":-125,"start":{"id":"ctrl"},"end":{"id":"w1"}},
 {"type":"arrow","id":"a2","x":180,"y":175,"width":160,"height":0,"start":{"id":"ctrl"},"end":{"id":"w2"},"label":{"text":"launch"}},
 {"type":"arrow","id":"a3","x":180,"y":190,"width":160,"height":125,"start":{"id":"ctrl"},"end":{"id":"w3"}},
 {"type":"arrow","id":"a4","x":670,"y":35,"width":-150,"height":0,"start":{"id":"data"},"end":{"id":"w1"},"label":{"text":"batches"}}
]}

2) Checkpoint without stopping for long

Each worker copies its shard of the training state to host memory (a few seconds of stall), then a background thread uploads it to object storage while training continues. A checkpoint counts only when every shard and a final manifest are written. The latest checkpoint also stays in host memory, replicated on a peer host, so most restarts load from memory instead of object storage.

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-80,"text":"Shaded: added in this step","fontSize":16},
 {"type":"rectangle","id":"ctrl","x":0,"y":140,"width":180,"height":70,"label":{"text":"Controller"}},
 {"type":"rectangle","id":"w1","x":340,"y":0,"width":180,"height":70,"label":{"text":"Workers 1"}},
 {"type":"rectangle","id":"w2","x":340,"y":140,"width":180,"height":70,"label":{"text":"Workers 2"}},
 {"type":"rectangle","id":"w3","x":340,"y":280,"width":180,"height":70,"label":{"text":"Workers N"}},
 {"type":"ellipse","id":"data","x":670,"y":-5,"width":220,"height":80,"label":{"text":"Training data"}},
 {"type":"rectangle","id":"snap","x":690,"y":140,"width":180,"height":70,"label":{"text":"Host memory\nsnapshot"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"ellipse","id":"store","x":1010,"y":135,"width":220,"height":80,"label":{"text":"Object store"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"arrow","id":"a1","x":180,"y":160,"width":160,"height":-125,"start":{"id":"ctrl"},"end":{"id":"w1"}},
 {"type":"arrow","id":"a2","x":180,"y":175,"width":160,"height":0,"start":{"id":"ctrl"},"end":{"id":"w2"},"label":{"text":"launch"}},
 {"type":"arrow","id":"a3","x":180,"y":190,"width":160,"height":125,"start":{"id":"ctrl"},"end":{"id":"w3"}},
 {"type":"arrow","id":"a4","x":670,"y":35,"width":-150,"height":0,"start":{"id":"data"},"end":{"id":"w1"},"label":{"text":"batches"}},
 {"type":"arrow","id":"a5","x":520,"y":175,"width":170,"height":0,"start":{"id":"w2"},"end":{"id":"snap"},"label":{"text":"shards"}},
 {"type":"arrow","id":"a6","x":870,"y":175,"width":140,"height":0,"start":{"id":"snap"},"end":{"id":"store"},"strokeStyle":"dashed"}
]}

3) Detect, replace, resume

Workers send heartbeats and per-step metrics to a health monitor. When a rank stops (hardware error, network timeout) or turns into a straggler, the monitor tells the controller, which cordons the host, swaps in a pre-warmed spare host, and restarts every rank from the latest complete checkpoint at the same data position. The monitor also watches loss and gradient norms for numerical trouble.

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-120,"text":"Shaded: added in this step. Dashed: async upload.","fontSize":16},
 {"type":"rectangle","id":"spares","x":0,"y":-20,"width":180,"height":70,"label":{"text":"Spare hosts"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"ctrl","x":0,"y":140,"width":180,"height":70,"label":{"text":"Controller"}},
 {"type":"rectangle","id":"mon","x":0,"y":320,"width":180,"height":70,"label":{"text":"Health monitor"},"backgroundColor":"#d0ebff","fillStyle":"solid"},
 {"type":"rectangle","id":"w1","x":340,"y":0,"width":180,"height":70,"label":{"text":"Workers 1"}},
 {"type":"rectangle","id":"w2","x":340,"y":140,"width":180,"height":70,"label":{"text":"Workers 2"}},
 {"type":"rectangle","id":"w3","x":340,"y":280,"width":180,"height":70,"label":{"text":"Workers N"}},
 {"type":"ellipse","id":"data","x":670,"y":-5,"width":220,"height":80,"label":{"text":"Training data"}},
 {"type":"rectangle","id":"snap","x":690,"y":140,"width":180,"height":70,"label":{"text":"Host memory\nsnapshot"}},
 {"type":"ellipse","id":"store","x":1010,"y":135,"width":220,"height":80,"label":{"text":"Object store"}},
 {"type":"arrow","id":"a1","x":180,"y":160,"width":160,"height":-125,"start":{"id":"ctrl"},"end":{"id":"w1"}},
 {"type":"arrow","id":"a2","x":180,"y":175,"width":160,"height":0,"start":{"id":"ctrl"},"end":{"id":"w2"},"label":{"text":"launch"}},
 {"type":"arrow","id":"a3","x":180,"y":190,"width":160,"height":125,"start":{"id":"ctrl"},"end":{"id":"w3"}},
 {"type":"arrow","id":"a4","x":670,"y":35,"width":-150,"height":0,"start":{"id":"data"},"end":{"id":"w1"},"label":{"text":"batches"}},
 {"type":"arrow","id":"a5","x":520,"y":175,"width":170,"height":0,"start":{"id":"w2"},"end":{"id":"snap"},"label":{"text":"shards"}},
 {"type":"arrow","id":"a6","x":870,"y":175,"width":140,"height":0,"start":{"id":"snap"},"end":{"id":"store"},"strokeStyle":"dashed"},
 {"type":"arrow","id":"a7","x":340,"y":335,"width":-160,"height":15,"start":{"id":"w3"},"end":{"id":"mon"}},
 {"type":"arrow","id":"a8","x":90,"y":320,"width":1,"height":-110,"start":{"id":"mon"},"end":{"id":"ctrl"},"label":{"text":"failures"}},
 {"type":"arrow","id":"a9","x":90,"y":50,"width":1,"height":90,"start":{"id":"spares"},"end":{"id":"ctrl"}}
]}

Final design

{"type":"excalidraw","elements":[
 {"type":"text","id":"legend","x":0,"y":-120,"text":"Dashed: async upload.","fontSize":16},
 {"type":"rectangle","id":"spares","x":0,"y":-20,"width":180,"height":70,"label":{"text":"Spare hosts"}},
 {"type":"rectangle","id":"ctrl","x":0,"y":140,"width":180,"height":70,"label":{"text":"Controller"}},
 {"type":"rectangle","id":"mon","x":0,"y":320,"width":180,"height":70,"label":{"text":"Health monitor"}},
 {"type":"rectangle","id":"w1","x":340,"y":0,"width":180,"height":70,"label":{"text":"Workers 1"}},
 {"type":"rectangle","id":"w2","x":340,"y":140,"width":180,"height":70,"label":{"text":"Workers 2"}},
 {"type":"rectangle","id":"w3","x":340,"y":280,"width":180,"height":70,"label":{"text":"Workers N"}},
 {"type":"ellipse","id":"data","x":670,"y":-5,"width":220,"height":80,"label":{"text":"Training data"}},
 {"type":"rectangle","id":"snap","x":690,"y":140,"width":180,"height":70,"label":{"text":"Host memory\nsnapshot"}},
 {"type":"ellipse","id":"store","x":1010,"y":135,"width":220,"height":80,"label":{"text":"Object store"}},
 {"type":"arrow","id":"a1","x":180,"y":160,"width":160,"height":-125,"start":{"id":"ctrl"},"end":{"id":"w1"}},
 {"type":"arrow","id":"a2","x":180,"y":175,"width":160,"height":0,"start":{"id":"ctrl"},"end":{"id":"w2"},"label":{"text":"launch"}},
 {"type":"arrow","id":"a3","x":180,"y":190,"width":160,"height":125,"start":{"id":"ctrl"},"end":{"id":"w3"}},
 {"type":"arrow","id":"a4","x":670,"y":35,"width":-150,"height":0,"start":{"id":"data"},"end":{"id":"w1"},"label":{"text":"batches"}},
 {"type":"arrow","id":"a5","x":520,"y":175,"width":170,"height":0,"start":{"id":"w2"},"end":{"id":"snap"},"label":{"text":"shards"}},
 {"type":"arrow","id":"a6","x":870,"y":175,"width":140,"height":0,"start":{"id":"snap"},"end":{"id":"store"},"strokeStyle":"dashed"},
 {"type":"arrow","id":"a7","x":340,"y":335,"width":-160,"height":15,"start":{"id":"w3"},"end":{"id":"mon"}},
 {"type":"arrow","id":"a8","x":90,"y":320,"width":1,"height":-110,"start":{"id":"mon"},"end":{"id":"ctrl"},"label":{"text":"failures"}},
 {"type":"arrow","id":"a9","x":90,"y":50,"width":1,"height":90,"start":{"id":"spares"},"end":{"id":"ctrl"}}
]}

How often to checkpoint

Checkpointing too often wastes time in stalls; too rarely loses more work per failure. With a stall of C seconds per checkpoint, a job mean time between failures M, and R seconds to restart, the first-order waste is:

waste(τ) = C / τ  +  (τ / 2 + R) / M

(stall share, plus on each failure half an interval of lost work and the restart). It is smallest at τ = sqrt(2 × C × M) (Young and Daly). With M ≈ 3 h = 10,986 s:

Two lessons: async snapshots matter, and restart time dominates once checkpoints are cheap. Every minute of restart costs about 8 × 1 = 8 minutes of the whole cluster a day. That is why spares are pre-warmed (container pulled, libraries loaded, network checked) and the latest checkpoint is kept in host memory.

import math
import random


def job_mtbf_hours(chips: int, chip_mtbf_hours: float) -> float:
    """Independent failures: the whole synchronous job stops when any chip fails."""
    return chip_mtbf_hours / chips


def optimal_interval(stall_s: float, mtbf_s: float) -> float:
    """Young/Daly first-order optimum: tau = sqrt(2 * C * M)."""
    return math.sqrt(2 * stall_s * mtbf_s)


def expected_waste(interval_s: float, stall_s: float, mtbf_s: float, restart_s: float) -> float:
    """Fraction of wall time not spent on useful steps (first-order model):
    checkpoint stalls, half an interval of lost work per failure, and restarts."""
    return stall_s / interval_s + (interval_s / 2 + restart_s) / mtbf_s


def simulate_goodput(interval_s: float, stall_s: float, mtbf_s: float, restart_s: float,
                     useful_target_s: float, seed: int = 0) -> float:
    """Monte Carlo check of the model: useful time / wall time."""
    rng = random.Random(seed)
    wall = useful = 0.0
    next_failure = rng.expovariate(1 / mtbf_s)
    while useful < useful_target_s:
        segment = interval_s + stall_s                 # work, then a checkpoint stall
        if wall + segment <= next_failure:
            wall += segment
            useful += interval_s
        else:
            wall = next_failure + restart_s            # lose the work since the last checkpoint
            next_failure = wall + rng.expovariate(1 / mtbf_s)
    return useful / wall

def test_failure_rate_and_optimal_interval():
    m = job_mtbf_hours(16384, 50000)
    assert abs(m - 3.05) < 0.01
    tau = optimal_interval(5, m * 3600)
    assert abs(tau - 331) < 1
    for worse in (tau / 2, tau * 2):          # the optimum really is a minimum
        assert expected_waste(worse, 5, m * 3600, 120) > expected_waste(tau, 5, m * 3600, 120)


def test_table_numbers():
    M = job_mtbf_hours(16384, 50000) * 3600
    rows = [(60, 600), (5, 600), (5, 120)]
    got = [round(expected_waste(optimal_interval(c, M), c, M, r), 3) for c, r in rows]
    assert got == [0.159, 0.085, 0.041]

def test_simulation_agrees_with_the_model():
    M = 3 * 3600
    for stall, restart in ((5, 120), (60, 600)):
        tau = optimal_interval(stall, M)
        model = expected_waste(tau, stall, M, restart)
        sim = 1 - simulate_goodput(tau, stall, M, restart, useful_target_s=3600 * 24 * 300, seed=1)
        assert abs(sim - model) < 0.02          # first-order model is slightly pessimistic

Making restarts exact

Resuming must reproduce the run, or debugging a loss curve becomes impossible:

• Data position. The loader is a pure function of (seed, step, data-parallel rank): a global shuffled order over shards, indexed by step. Saving step is enough to resume; no loader state to snapshot. Changing the number of data-parallel groups after a failure needs the same global order, re-split.
• Random state. Save RNG states (dropout, sampling) with the checkpoint.
• Atomic checkpoints. Write shards, verify checksums, then write the manifest last. A crash mid-checkpoint leaves the previous one valid. Keep the last few, plus daily ones for rollbacks.

Detecting failures, stragglers and bad numbers

• Hard failures: missed heartbeats, collective-operation timeouts, hardware error counters (memory errors, link errors). Detection within a minute matters as much as restart speed; set collective timeouts tight (minutes, not the default half hour).
• Stragglers: synchronous training runs at the speed of the slowest chip. Track per-rank step time; a rank consistently 5% slower (thermal throttling, a bad link) gets drained at the next checkpoint and replaced.
• Loss spikes and NaNs: on a NaN or a loss far above its moving average, stop, roll back to the checkpoint before the spike, skip the batches that caused it (or lower the learning rate briefly), and continue. Automate this for the common case; page a person if it repeats.
• Silent data corruption: a chip that computes wrong answers without raising errors is the most dangerous failure, since it poisons the weights quietly. Detect it by comparing data-parallel replicas (parameters should match bit-for-bit after each optimizer step; compare checksums every N steps), by per-replica gradient-norm outliers, and by periodically re-running a step on a spare and comparing.

Parallelism layout and failures

Tensor parallelism stays inside a node so its heavy traffic uses the fastest links; pipeline stages span nodes in the same network pod; data-parallel replicas go across pods. A failed host takes out one tensor group of one pipeline stage of one replica, but because training is synchronous, everything waits. Elastic alternatives (continue with one fewer data-parallel replica, re-shard the optimizer state, adjust batch size or learning rate) avoid waiting for a spare but change the training dynamics, so most frontier runs prefer spares and an exact restart.

Observability

One dashboard per run: loss and gradient norm, tokens per second, model FLOPs utilization, goodput broken down into stalls, lost work and restart time, incidents per day with root causes, and checkpoint age. Goodput by cause tells you which investment (faster restarts, better hardware screening, fewer spikes) buys the most.

• Checkpoint interval: shorter loses less on failure but stalls more; the formula and the real numbers decide, and they change as the job grows.
• Spares versus elastic training: spares keep the run exact but idle hardware (2 to 3% of hosts); elastic training uses every chip but changes the batch size or sharding mid-run.
• Peer-memory checkpoints versus object storage only: memory restarts are fast but lost if both a host and its peer fail; object storage is the durable fallback.
• Aggressive health checks versus false alarms: draining slow or suspicious hosts early prevents hours of degraded throughput, at the cost of sometimes replacing healthy ones.

Mid-level (L4)

• Explains data, tensor and pipeline parallelism and why a single failure stops synchronous training.
• Periodic checkpoints and restart from the latest one, with a resumable data loader.
• Knows that failures are frequent at this scale.

Senior (L5)

• Works out the failure rate, the training state size per chip and the checkpoint interval with the Young/Daly formula, with numbers.
• Async snapshots to host memory, atomic manifests, exact data and RNG resumption, hot spares.
• Handles stragglers and loss spikes with automated rollback.

Staff (L6+)

• Shows that restart time dominates once checkpoints are cheap, and designs for minutes-level recovery (pre-warmed spares, peer-memory checkpoints, fast detection).
• Addresses silent data corruption with replica comparison and recomputation.
• Treats goodput by cause as the run's key metric and weighs elastic training against exact restarts.