Why checkpointing sets the design
A long training run will lose nodes. The standard defence is to save the full model and optimiser state periodically so a failure costs one interval rather than the whole run. That state is large, every node writes at once, and everything stops while it happens — so checkpoint write bandwidth translates directly into wasted accelerator time.