Design review: checkpointing a large training run

You are reviewing the checkpointing design for a month-long training run. Critique it: what fails, what it costs, what you would change, and what the doc does not cover. You have 45 minutes.

Checkpointing for the 70B run (draft)

Setup. A 70B-parameter model trained on 1,024 GPUs (128 nodes) for 30 days with mixed precision (bf16 compute) and the Adam optimizer, sharded across ranks.

Size. A checkpoint is the model weights: 70B × 2 bytes ≈ 140 GB.

Schedule. Every 2 hours, all ranks stop training and write their shards to one shared NFS server (about 5 GB/s of write bandwidth).

Commit. Rank 0 first updates the latest file to point at the new checkpoint directory, then every rank writes its shard there.

Restart. After a failure the job restarts from latest. The data loader restarts from the beginning of the current epoch (one epoch is about 6 days).

Retention. All checkpoints are kept for the life of the project.

Topology. Restoring requires the same 1,024-GPU layout, because each shard maps to one rank.

Validation. None; NFS writes are reliable.