A full checkpoint for a 70B model includes: model weights (~140 GB in FP16), optimizer states (~280 GB for Adam, which stores two moving averages per parameter), learning rate scheduler state, random number generator states, and the current training step. Total: ~420 GB per checkpoint. Saving this to disk takes significant time and storage, which is why checkpointing is done periodically rather than every step.
Common strategies: save every N steps (simple but uses lots of storage), save only the K most recent checkpoints (deleting older ones to save space), save based on evaluation metrics (keep the checkpoint with the best validation loss), and use async checkpointing (save in the background while training continues on the next batch). Large training runs often use all of these: frequent local checkpoints on fast NVMe storage plus periodic remote checkpoints to network storage for disaster recovery.
Different frameworks use different checkpoint formats: PyTorch's state_dict, Hugging Face's safetensors, FSDP's sharded checkpoints, and DeepSpeed's ZeRO checkpoints. Converting between formats is a common task — you might train with DeepSpeed (sharded across GPUs) but need a single consolidated checkpoint for inference or uploading to Hugging Face. The safetensors format is becoming the standard for sharing because it's fast to load and memory-safe.