Filtered by: LLM × Clear all

Elastic Gang: Per-Token Membership Change for a Hard-Barriered LLM Inference Gang Co-Scheduled with OS Processes

Daeyeon Son 2026-07-07
Inference LLM × Scheduling

The problem is that on-device LLM decoding is a hard-barriered CPU-SIMD computation requiring all cores per token, but preemptive scheduling causes deadlock or silent logit corruption. The method introduces the elastic gang in Anima OS, using an ACK-latched epoch protocol with generation-tagged latches and RCU-style membership consent to allow per-token core membership changes without waiting on named cores. Experimental evidence on AMD Zen 5 shows bit-exact inference under per-token membership changes for 135M and 7B models, with 1.75x, 1.52x, and 1.28x general throughput improvements over static 8-core splits at 25%, 50%, and 75% inference duty cycles, and core return costs of 0.22 us. This matters because it enables safe, dynamic core sharing between LLM inference and OS processes, Pareto-dominating static partitions and auto-sizing the gang online to maximize throughput without stranded cores.

PDF

ELiTeFormer: An Efficient Transformer for FPGAs

Victor Agostinelli, Nicolas Bohm Agostini, Antonino Tumeo 2026-07-07
Attention Language Model LLM × Quantization Transformer Fpga GPU

ELiTeFormer addresses the deployment challenge of Transformer blocks in LLMs by co-designing hybrid linear attention with ultra-low-precision ternary projections specifically for FPGAs. The method introduces a novel processing element micro-architecture that eliminates all multiplications in ternary linear projections using bitmasking, avoiding DSP blocks entirely. Experimental results show 10x model weight compression and 12.8x KV cache compression versus LLaMA 3, with block-level simulations achieving 9.6x speedup for FFN and 4.4x for attention, and end-to-end deployment delivering up to 3.9x lower latency and 3.2x better energy efficiency than an NVIDIA A100 GPU. This matters as it demonstrates the first FPGA realization combining linear attention with ternary quantization, proving algorithm-architecture co-design viability for next-generation LLM acceleration.

PDF

Direct Model State Migration for Elastic Training of Large Language Models

Weijian Liu, Mingzhen Li, Rui Kang, Chen Sun 2026-07-07
Language Model LLM × Training GPU

The problem is that elastic training of large language models (LLMs) requires state migration across device sets when hybrid-parallel configurations change, but existing checkpoint-based solutions force all GPUs to stall and incur prohibitive latency from data movement across memory hierarchies. ETC proposes a checkpoint-free state migration framework that exploits state locality to minimize inter-GPU data movement, replacing storage persistence with direct peer-to-peer communication and eliminating node fragmentation through communication coalescing. Integrated with Megatron-LM, ETC reduces migration overhead by 2.33× to 6.37× compared to checkpoint-based solutions across diverse parallel configurations. This matters because ETC unlocks practical elastic training in production environments by enabling efficient migration, allowing LLM training to adapt to dynamic resources in shared clusters.

PDF