Direct Model State Migration for Elastic Training of Large Language Models

The problem is that elastic training of large language models (LLMs) requires state migration across device sets when hybrid-parallel configurations change, but existing checkpoint-based solutions force all GPUs to stall and incur prohibitive latency from data movement across memory hierarchies. ETC proposes a checkpoint-free state migration framework that exploits state locality to minimize inter-GPU data movement, replacing storage persistence with direct peer-to-peer communication and eliminating node fragmentation through communication coalescing. Integrated with Megatron-LM, ETC reduces migration overhead by 2.33× to 6.37× compared to checkpoint-based solutions across diverse parallel configurations. This matters because ETC unlocks practical elastic training in production environments by enabling efficient migration, allowing LLM training to adapt to dynamic resources in shared clusters.