Filtered by: Inference × Clear all

Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent

Lingyun Yang, Yuxiao Wang, Shenghao Liang, Linfeng Yang 2026-07-19
Inference × LLM GPU Hardware Agent

Atrex-Bench addresses the problem that existing GPU kernel benchmarks use synthetic or curated workloads, not production traces. The method samples 30 operators and 440 shapes from full-cluster inference traces, weighting each by GPU time and card-hours. Experimental evidence shows the best vanilla model reaches only ~10% of roofline performance, with much apparent correctness from PyTorch fallbacks. This matters because the co-released Atrex-Kernel-Agent (AKA) converts fallbacks into kernels matching hand-tuned baselines, demonstrating a path to production-ready LLM-generated kernels.

PDF

ExaGEMM: Exploration Framework for CPU-Driven ML Inference via Associative In-Register Computing for Low-Bit GEMM

Hyunwoo Oh, Suyeon Jang, Hanning Chen, Sanggeon Yun 2026-07-19
Gem5 Microarchitecture Simulation Inference ×

The problem is that low-bit GEMM is central to efficient ML inference, but very-low-bit execution is poorly suited for conventional CPUs, and fragmented precision regimes make lightweight CPU support selection a first-class design challenge. ExaGEMM proposes a workload-aware codesign framework for CPU-native low-bit GEMM via register-resident LUT execution, requiring only an in-register select/feed mechanism with explicitly modeled cost. Experimental evidence shows ExaGEMM improves latency by 13.29x over software-only baselines across representative ML models and CPU targets, while pruning the candidate space by 99.2% before simulation. This matters because it demonstrates that workload-aware frontier selection is especially important for mixed-precision LLM workloads, enabling efficient CPU-driven ML inference without major hardware redesign.

PDF

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao 2026-07-19
Attention Inference × Training GPU

LongStraw addresses the growing gap between inference context lengths and RL post-training, where inference systems handle million-token contexts but post-training often stays at 256K tokens or below. The method evaluates shared prompts without autograd, retains only model-specific state for later tokens, and replays short response branches one at a time to reduce the live training graph under a fixed GPU budget. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8, with a stress test reaching 4.46M positions, while on 32 H20 GPUs it validates the execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. This matters because it enables million-token RL post-training for AI agents with long trajectories under fixed GPU budgets, though the abstract does not disclose complete training correctness.

PDF

PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

Hyunwoo Oh, Suyeon Jang, Hanning Chen, KyungIn Nam 2026-07-19
Inference × LLM Quantization Compiler Low Bit Runtime

PolyQ addresses the problem that existing low-bit quantization for CPU LLM inference offers either coarse operating points or fine-grained mixed precision that is inefficient on CPUs. The method is a compiler/quantization co-design that assigns per-channel bit-widths from {2,3,4,8,16} and uses compile-time permutation and clustering to generate SIMD- and LUT-compatible kernels with layout regularization off the runtime path. On Falcon-H1-3B, Llama2-13B, and Qwen3-32B, PolyQ improves perplexity by 2.4–32.1% over prior methods at a 3b target and reduces activation reorder traffic by up to 70.8% on three representative CPUs. This matters because it demonstrates that fractional-bit CPU deployment is practical, predictable, and energy-efficient for scalable edge inference.

PDF

NIFA: Nonlinear IMC enhanced FPGA for efficient ML inference

Jiajun Hu, Ruthwik Reddy Sunketa, Lei Zhao, Archit Gajjar 2026-07-19
Design Space Exploration Agentic Architecture Inference ×

The problem is that conventional ReRAM-based IMC blocks in FPGAs only support static-weight VMM, limiting efficiency gains for Transformer models that require nonlinear and dynamic matrix-matrix multiplication (DIMM), while ADCs consume over 70% of IMC block area and power. The method proposes a novel FPGA architecture with an ADC-free IMC block using analog content-addressable memories (ACAMs) for native nonlinear operations, along with FPGA-aware design-space exploration and efficient mapping for DIMM. Experimental evidence shows up to 40x and 1.9x higher energy efficiency and 4.1x and 2.5x higher area efficiency on CNN and Transformer benchmarks, respectively. This matters because it significantly improves FPGA DL inference efficiency, especially for Transformer-based workloads, advancing domain-specialized FPGA design.

PDF