Daily | Yixun Hong

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao 2026-07-20

Attention Inference × Training GPU

LongStraw addresses the growing gap between inference context lengths and RL post-training, where inference systems handle million-token contexts but post-training typically stays at 256K tokens or below, especially critical for AI agents with long trajectories. The method is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, using GRPO to evaluate shared prompts without autograd, retain only model-specific state, and replay short response branches one at a time to reduce the live training graph. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8, with only 0.21 GB added peak memory per group size increase, and a separate stress test reaches 4.46M positions; on 32 H20 GPUs, it validates the end-to-end execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. This matters because it demonstrates execution capacity for million-token RL post-training under fixed GPU budgets, though the abstract does not disclose complete training correctness due to detached prompt state and incomplete distributed paths.

PDF

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam 2026-07-20

Inference × LLM Quantization Compiler Low Bit Runtime

PolyQ addresses the problem that existing low-bit quantization for CPU-based LLM inference offers either coarse operating points or fine-grained mixed precision that is inefficient to execute. The method is a compiler/quantization co-design that assigns per-channel bit-widths from {2,3,4,8,16} and uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks with SIMD- and LUT-compatible kernels. On Falcon-H1-3B, Llama2-13B, and Qwen3-32B, PolyQ improves perplexity by 2.4–32.1% over prior methods at a 3-bit target, and compiler layout regularization reduces activation reorder traffic by up to 70.8% on three representative CPUs. This matters because it demonstrates that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets, enabling scalable on-device LLM inference.

PDF

ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

Hyunwoo Oh, Suyeon Jang, Hanning Chen, Sanggeon Yun 2026-07-20

Gem5 Microarchitecture Simulation Inference ×

The problem is that low-bit GEMM is central to efficient ML inference, but very-low-bit execution is poorly suited to conventional CPUs, and the fragmented design space of precision regimes makes lightweight CPU support selection a first-class challenge. ExaGEMM proposes a workload-aware codesign framework for CPU-native low-bit GEMM using register-resident LUT execution, requiring only an in-register select/feed mechanism as new hardware. Experimental evidence shows ExaGEMM improves latency by 13.29x over software-only baselines across representative ML models and CPU targets, while pruning the candidate space by 99.2% before simulation. This matters because it enables practical, efficient CPU-driven ML inference for mixed-precision LLM workloads, demonstrating that workload-aware frontier selection is critical for low-bit GEMM support.

PDF

NIFA: Nonlinear IMC enhanced FPGA for efficient ML inference

Jiajun Hu, Ruthwik Reddy Sunketa, Lei Zhao, Archit Gajjar 2026-07-20

Design Space Exploration Agentic Architecture Inference ×

The problem is that conventional ReRAM-based IMC blocks in FPGAs only support static-weight VMM, limiting efficiency gains for Transformer models that require nonlinear and dynamic matrix-matrix multiplication (DIMM) operations, while ADCs consume over 70% of IMC block area and power. The method proposes NIFA, a novel FPGA architecture integrating an ADC-free IMC block that replaces ADCs with analog content-addressable memories (ACAMs) to natively perform nonlinear operations, combined with FPGA-aware design-space exploration and an efficient mapping for DIMM. Experimental evidence shows up to 40x and 1.9x higher energy efficiency and 4.1x and 2.5x higher area efficiency on CNN and Transformer-based benchmarks, respectively. This matters because it significantly improves FPGA DL inference efficiency and sustains robust gains on Transformer-based workloads, advancing domain-specialized FPGA design for modern ML models.

PDF