Infrastructure & Agents

What Is the Zero Redundancy Optimizer (ZeRO)?

The Zero Redundancy Optimizer splits optimizer state, gradients, and parameters across devices instead of duplicating them. This reduces memory use per device and enables training of very large models.