Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts
The problem is that conventional Mixture-of-Experts (MoE) implementations launch the return all-to-all communication after expert compute completes, exposing communication latency on the critical path and reducing GPU utilization. The method introduces a fine-grained producer-consumer co-design with tile-level signaling and scheduling, using a persistent per-rank computation kernel and a persistent communication kernel on a small dedicated SM partition to overlap expert compute with the second all-to-all. On a 4-A100 GPU platform, evaluated on three MoE models against four state-of-the-art MoE systems, the approach achieves up to 2.64x end-to-end speedup and 2.74x MoE-layer speedup. This matters because it provides a practical, non-intrusive way to improve distributed MoE execution efficiency on multi-GPU systems, enabling better scaling of large language models.