Filtered by: Language × Clear all

Fine-grained Computation-Communication Overlap via Tile-level Signaling and Scheduling for Mixture-of-Experts

Minyu Cui, Anna Wingkvist, Morgan Ericsson 2026-07-24
Language × Model GPU Communication Scheduling

The problem is that conventional Mixture-of-Experts (MoE) implementations launch the return all-to-all communication after expert compute completes, exposing communication latency on the critical path and reducing GPU utilization. The method introduces a fine-grained producer-consumer co-design with tile-level signaling and scheduling, using a persistent per-rank computation kernel and a persistent communication kernel on a small dedicated SM partition to overlap expert compute with the second all-to-all. On a 4-A100 GPU platform, evaluated on three MoE models against four state-of-the-art MoE systems, the approach achieves up to 2.64x end-to-end speedup and 2.74x MoE-layer speedup. This matters because it provides a practical, non-intrusive way to improve distributed MoE execution efficiency on multi-GPU systems, enabling better scaling of large language models.

PDF

Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing

Sadegh Majidi, Niloofar Mireshghallah, Kazem Taram 2026-07-24
Attention Inference Language × Model Transformer Hardware Architecture

LeakyLMs introduces a set of attacks that leak proprietary model architecture and deployment information from production language models using only per-token generation timing. The method builds a detailed timing model of token generation on NVIDIA GPUs and performs a search over the architecture space to infer properties like transformer layers, hidden dimension size, and attention heads. Experimental evidence shows that for Llama models, the near-correct architectural configuration appears in the top-10 guesses over 90% of the time, and the attack successfully detects speculative decoding in Google Gemini Flash 2.5 with a draft context window of approximately 128K tokens. This matters because it demonstrates that sensitive model and deployment details can be stolen remotely via timing side channels, posing a significant security risk to proprietary language model services.

PDF